Page MenuHomePhabricator
Create Task
Maniphest T203075

Warning: MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Seems something is corrupted on some wiki

5 MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row 5191150. [Called from MediaWiki\Storage\SqlBlobStore::fetchBlob in /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/SqlBlobStore.php

Triggering URL: https://de.wikipedia.org/w/index.php?title=Ahmadiyya&limit=500&action=history

Logstash: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.08.29/mediawiki/?id=AWWFXFYy8VFZOIjHiQmC

Logstash search query: "Bad data in text row" AND type:"mediawiki"

Stacktrace:

#0 /srv/mediawiki/php-1.32.0-wmf.18/includes/debug/MWDebug.php(309): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.18/includes/debug/MWDebug.php(164): MWDebug::sendMessage(string, array, string, integer)
#2 /srv/mediawiki/php-1.32.0-wmf.18/includes/GlobalFunctions.php(1148): MWDebug::warning(string, integer, integer, string)
#3 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/SqlBlobStore.php(356): wfLogWarning(string)
#4 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/SqlBlobStore.php(281): MediaWiki\Storage\SqlBlobStore->fetchBlob(string, integer)
#5 /srv/mediawiki/php-1.32.0-wmf.18/includes/libs/objectcache/WANObjectCache.php(1246): Closure$MediaWiki\Storage\SqlBlobStore::getBlob(boolean, integer, array, NULL)
#6 /srv/mediawiki/php-1.32.0-wmf.18/includes/libs/objectcache/WANObjectCache.php(1119): WANObjectCache->doGetWithSetCallback(string, integer, Closure$MediaWiki\Storage\SqlBlobStore::getBlob;515, array)
#7 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/SqlBlobStore.php(284): WANObjectCache->getWithSetCallback(string, integer, Closure$MediaWiki\Storage\SqlBlobStore::getBlob;515, array)
#8 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/RevisionStore.php(1382): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#9 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/RevisionStore.php(1319): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#10 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
#11 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/SlotRecord.php(306): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;510, MediaWiki\Storage\SlotRecord)
#12 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/SlotRecord.php(512): MediaWiki\Storage\SlotRecord->getContent()
#13 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/RevisionSlots.php(149): MediaWiki\Storage\SlotRecord->getSize()
#14 [internal function]: Closure$MediaWiki\Storage\RevisionSlots::computeSize(integer, MediaWiki\Storage\SlotRecord)
#15 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/RevisionSlots.php(150): array_reduce(array, Closure$MediaWiki\Storage\RevisionSlots::computeSize;603, integer)
#16 /srv/mediawiki/php-1.32.0-wmf.18/includes/Storage/RevisionStoreRecord.php(160): MediaWiki\Storage\RevisionSlots->computeSize()
#17 /srv/mediawiki/php-1.32.0-wmf.18/includes/Revision.php(707): MediaWiki\Storage\RevisionStoreRecord->getSize()
#18 /srv/mediawiki/php-1.32.0-wmf.18/includes/actions/HistoryAction.php(726): Revision->getSize()
#19 /srv/mediawiki/php-1.32.0-wmf.18/includes/actions/HistoryAction.php(486): HistoryPager->historyLine(stdClass, stdClass, boolean, boolean, boolean)
#20 /srv/mediawiki/php-1.32.0-wmf.18/includes/pager/IndexPager.php(439): HistoryPager->formatRow(stdClass)
#21 /srv/mediawiki/php-1.32.0-wmf.18/includes/actions/HistoryAction.php(236): IndexPager->getBody()
#22 /srv/mediawiki/php-1.32.0-wmf.18/includes/actions/FormlessAction.php(43): HistoryAction->onView()
#23 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(501): FormlessAction->show()
#24 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(294): MediaWiki->performAction(Article, Title)
#25 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(868): MediaWiki->performRequest()
#26 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(525): MediaWiki->main()
#27 /srv/mediawiki/php-1.32.0-wmf.18/index.php(42): MediaWiki->run()
#28 /srv/mediawiki/w/index.php(3): include(string)
#29 {main}

Related Objects
Search...

Event Timeline

hashar created this task.Aug 29 2018, 11:11 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 29 2018, 11:11 AM
Addshore updated the task description. (Show Details)Aug 29 2018, 12:51 PM
Aklapper added a project: Multi-Content-Revisions.Aug 29 2018, 1:44 PM

Uneducatedly adding Multi-Content-Revisions as the stacktrace mentions RevisionStore and emulateMainSlot. Feel very free to correct.

Krinkle added a project: Core-Platform-Team-Old.Sep 1 2018, 1:25 AM
Krinkle subscribed.

Might be related to T203255. Tagging with CPT for further investigation.

Krinkle moved this task from Untriaged to Dec2019/1.35.wmf.10+ on the Wikimedia-production-error board.Sep 1 2018, 1:25 AM
Anomie subscribed.Sep 4 2018, 1:38 PM
In T203075#4550040, @Krinkle wrote:

Might be related to T203255.

Considering ApiComparePages isn't in the stack trace, that seems very unlikely.

Tagging with CPT for further investigation.

The Multi-Content-Revisions tagging seems more likely relevant to me, although in the end it looks like it's really an instance of T22757: Corruption of text from early 2005 due to HistoryBlobStub pointers broken by recompressTracked.php.

This specific error seems to be due to a revision from 2005: https://de.wikipedia.org/w/index.php?title=Ahmadiyya&oldid=5191150. That revision has a HistoryBlobStub pointing to revision 5810513 with a hash of 823ec0d61c9dc4dad1029cc30c884892, but 5810513's ConcatenatedGzipHistoryBlob doesn't have an item with that hash.

If I ever get to get back to T181555: Remove use of PHP serialization in revision storage, the migration script for that will wind up fixing these broken references by flagging them as errors instead.

Anomie mentioned this in T175313: SqlBlobStore emits "PHP Warning: data error" from gzinflate().Sep 4 2018, 1:57 PM
CCicalese_WMF moved this task from Inbox to CPT TEC1 Backlog on the Core-Platform-Team-Old board.Sep 5 2018, 1:25 AM
CCicalese_WMF removed a project: Multi-Content-Revisions.Sep 7 2018, 12:56 PM
Krinkle moved this task from Dec2019/1.35.wmf.10+ to Older on the Wikimedia-production-error board.Sep 16 2018, 7:04 PM
Krinkle mentioned this in T116455: Commons page/file archive corruption of File:Wikimedia-servers-2004-11-13.png .Sep 16 2018, 7:16 PM
Krinkle moved this task from Older to Apr 2019 / 1.33.wmf.25+ on the Wikimedia-production-error board.
Krinkle moved this task from Apr 2019 / 1.33.wmf.25+ to Older on the Wikimedia-production-error board.Sep 26 2018, 2:26 AM
CCicalese_WMF added a project: Platform Team Legacy (Next).Oct 1 2018, 11:23 PM
CCicalese_WMF edited projects, added Core Platform Team Goals (Security, stability, performance and scalability (TEC1)); removed Core-Platform-Team-Old.Oct 2 2018, 12:06 AM
CCicalese_WMF edited projects, added Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)); removed Core Platform Team Goals (Security, stability, performance and scalability (TEC1)).Oct 2 2018, 2:24 AM
Krinkle mentioned this in T108495: Special:Undelete not opening for some titles due to MediaWiki\Storage\RevisionAccessException.Oct 2 2018, 8:34 PM
Krinkle added a subscriber: ArielGlenn.Oct 3 2018, 2:00 AM

Noticed a sudden burst of the following in the logs. Same error as this task, although with a different stack.

channel: error
cli_argv: /srv/mediawiki/php-1.32.0-wmf.23/../multiversion/MWScript.php fetchText.php --wiki hrwiki
exception_id: afd331d39d8f662991b7b7c8
host: snapshot1009

PHP Warning: Revision::getRevisionText: Bad data in text row!

#1 /srv/mediawiki/php-1.32.0-wmf.23/includes/debug/MWDebug.php(309): trigger_error(string, integer)
#2 /srv/mediawiki/php-1.32.0-wmf.23/includes/debug/MWDebug.php(164): MWDebug::sendMessage(string, array, string, integer)
#3 /srv/mediawiki/php-1.32.0-wmf.23/includes/GlobalFunctions.php(1147): MWDebug::warning(string, integer, integer, string)
#4 /srv/mediawiki/php-1.32.0-wmf.23/includes/Revision.php(1073): wfLogWarning(string)
#5 /srv/mediawiki/php-1.32.0-wmf.23/maintenance/fetchText.php(86): Revision::getRevisionText(stdClass)
#6 /srv/mediawiki/php-1.32.0-wmf.23/maintenance/fetchText.php(63): FetchText->doGetText(Wikimedia\Rdbms\DatabaseMysqli, integer)
#7 /srv/mediawiki/php-1.32.0-wmf.23/maintenance/doMaintenance.php(94): FetchText->execute()
#8 /srv/mediawiki/php-1.32.0-wmf.23/maintenance/fetchText.php(96): require_once(string)
#9 /srv/mediawiki/multiversion/MWScript.php(100): require_once(string)

@Anomie Is this the same root cause as the one we see from "History" views in prod?

@ArielGlenn Do you know if there are any side-effect or impact from this error that we need to be aware of? Or does the snapshot script account for this issue and recovers without issue?

Apologies if this was already discussed previously, but I'm somewhat surprised to see a maintenance script run in Eqiad. In theory snapshot scripts are read-only which should be safe in Eqiad even when Codfw is primary, but MediaWiki isn't ready for active-active, which by default I'd assume means read operations are unsafe in Eqiad, because MediaWiki has side-effects on read operations (in particular around caching and job queue).

ArielGlenn added a comment.Oct 3 2018, 7:20 AM
In T203075#4637351, @Krinkle wrote:

...

@ArielGlenn Do you know if there are any side-effect or impact from this error that we need to be aware of? Or does the snapshot script account for this issue and recovers without issue?

The script will move on to the next revision; we expect and account for buglets like these. Thanks for checking though!

Apologies if this was already discussed previously, but I'm somewhat surprised to see a maintenance script run in Eqiad. In theory snapshot scripts are read-only which should be safe in Eqiad even when Codfw is primary, but MediaWiki isn't ready for active-active, which by default I'd assume means read operations are unsafe in Eqiad, because MediaWiki has side-effects on read operations (in particular around caching and job queue).

It probably has been discussed elsewhere, but I'll summarize here: there has never been any plan to have dumps generation hosts in any other data center; they are in eqiad only. It's been this way for our past datacenter switchovers too. Note that, since Mediawiki scripts look at /etc/cluster to see which db config file to use, the fact that these are run from hosts in eqiad means that they use eqiad db servers; of course all of these requests are reads only.

Anomie added a comment.Oct 3 2018, 2:44 PM
In T203075#4637351, @Krinkle wrote:

@Anomie Is this the same root cause as the one we see from "History" views in prod?

Probably. There's no rev_id, text_id, or the like in that error message so it's hard to tell for sure.

daniel subscribed.Mar 26 2019, 2:57 PM

Is this still happening? If not, can this ticket be closed?

• kchapman moved this task from Next to Watching / External on the Platform Team Legacy board.Mar 26 2019, 2:57 PM
• kchapman edited projects, added Platform Team Legacy (Watching / External); removed Platform Team Legacy (Next).
hashar added a comment.Apr 3 2019, 7:36 AM

Yes that is still occurring. They can be found in logstash with the search query: "Bad data in text row" AND type:"mediawiki"

hashar updated the task description. (Show Details)Apr 3 2019, 7:36 AM
ArielGlenn added a comment.Apr 4 2019, 11:15 PM

Yep, I see a batch of them.. Here's one from today for hrwiki:

PHP Warning: MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row 1677927. [Called from MediaWiki\Storage\SqlBlobStore::fetchBlob in /srv/mediawiki/php-1.33.0-wmf.23/includes/Storage/SqlBlobStore.php at line 353]

It's being triggered by a stubs run, which should not fetch revision text, so that concerns me quite a bit. That's new behavior since the March 20th run, and a regression of some kind, but I'll open another task for that.

ArielGlenn mentioned this in T220160: getRedirectTarget should not automatically load revision content in all cases.Apr 4 2019, 11:23 PM
ArielGlenn added a comment.Apr 10 2019, 9:56 AM

https://hr.wikipedia.org/wiki/Franjeva%C4%8Dki_samostan_Sv._Antuna_Padovanskog_u_Koprivnici This url triggers the exception. Logstash entry: https://logstash.wikimedia.org/goto/2fc2e8da6b9cab496c8eb409b7347b21 Here's the relevant info from the page table, revision table for the current revision, and text table for that text id.

wikiadmin@10.64.0.205(hrwiki)> select * from page where page_id = 192386;
+---------+----------------+-----------------------------------------------------------+-------------------+------------------+-------------+----------------+----------------+--------------------+-------------+----------+--------------------+-----------+
| page_id | page_namespace | page_title                                                | page_restrictions | page_is_redirect | page_is_new | page_random    | page_touched   | page_links_updated | page_latest | page_len | page_content_model | page_lang |
+---------+----------------+-----------------------------------------------------------+-------------------+------------------+-------------+----------------+----------------+--------------------+-------------+----------+--------------------+-----------+
|  192386 |              0 | Franjevački_samostan_Sv._Antuna_Padovanskog_u_Koprivnici  |                   |                1 |           1 | 0.222437823375 | 20160904152621 | NULL               |     1705637 |       81 | wikitext           | NULL      |
+---------+----------------+-----------------------------------------------------------+-------------------+------------------+-------------+----------------+----------------+--------------------+-------------+----------+--------------------+-----------+
1 row in set (0.00 sec)

wikiadmin@10.64.0.205(hrwiki)> select * from revision where rev_id = 1705637;
+---------+----------+-------------+-------------+----------+---------------+----------------+----------------+-------------+---------+---------------+----------+-------------------+--------------------+
| rev_id  | rev_page | rev_text_id | rev_comment | rev_user | rev_user_text | rev_timestamp  | rev_minor_edit | rev_deleted | rev_len | rev_parent_id | rev_sha1 | rev_content_model | rev_content_format |
+---------+----------+-------------+-------------+----------+---------------+----------------+----------------+-------------+---------+---------------+----------+-------------------+--------------------+
| 1705637 |   192386 |     1677927 |             |    17775 | Fhms          | 20090309211443 |              0 |           0 |      81 |             0 |          | NULL              | NULL               |
+---------+----------+-------------+-------------+----------+---------------+----------------+----------------+-------------+---------+---------------+----------+-------------------+--------------------+
1 row in set (0.00 sec)

wikiadmin@10.64.0.205(hrwiki)> select * from text where old_id = 1677927;
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| old_id  | old_namespace | old_title | old_text         | old_comment | old_user | old_user_text | old_timestamp | old_minor_edit | old_flags           | inverse_timestamp |
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| 1677927 |               |           | DB://cluster20/0 |             |          |               |               |                | utf-8,gzip,external |                   |
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
1 row in set (0.00 sec)

We should do something about entries in the text table like this one that obviously points to crap. But I don't know what that something is.

hashar unsubscribed.Apr 10 2019, 12:05 PM
ArielGlenn added a comment.EditedApr 10 2019, 2:04 PM

https://logstash.wikimedia.org/goto/57a4c9ec510eb287f8f7d48a3db08f66 Another sample, these turned up in the abstract dumps for jvwiki. There's a workaround patch for that in wmf.25 which will mask the problem but the bad data will still be there.

wikiadmin@10.64.16.191(jvwiki)> select * from text where old_id = 209199;
+--------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| old_id | old_namespace | old_title | old_text         | old_comment | old_user | old_user_text | old_timestamp | old_minor_edit | old_flags           | inverse_timestamp |
+--------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| 209199 |               |           | DB://cluster20/0 |             |          |               |               |                | utf-8,gzip,external |                   |
+--------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
Krinkle mentioned this in T205936: Unable to view some pages due to fatal RevisionAccessException: "Failed to load data blob from tt".Jun 9 2019, 4:17 PM
CCicalese_WMF removed a project: Platform Team Legacy (Watching / External).Jul 26 2019, 5:47 PM
brennen subscribed.Aug 9 2019, 2:18 PM

Seeing 3659 of these (or pretty similar) since 03:30 UTC:

Triggering URL: https://zh.wikipedia.org/wiki/Delon_Thamrin

PHP Warning: MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row 9375723. [Called from MediaWiki\Storage\SqlBlobStore::fetchBlob in /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php at line 361]
#0 /srv/mediawiki/php-1.34.0-wmf.17/includes/debug/MWDebug.php(309): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.34.0-wmf.17/includes/debug/MWDebug.php(164): MWDebug::sendMessage(string, array, string, integer)
#2 /srv/mediawiki/php-1.34.0-wmf.17/includes/GlobalFunctions.php(1078): MWDebug::warning(string, integer, integer, string)
#3 /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php(361): wfLogWarning(string)
#4 /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlob(string, integer)
#5 /srv/mediawiki/php-1.34.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1417): Closure$MediaWiki\Storage\SqlBlobStore::getBlob(boolean, integer, array, NULL)
#6 /srv/mediawiki/php-1.34.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1271): WANObjectCache->fetchOrRegenerate(string, integer, Closure$MediaWiki\Storage\SqlBlobStore::getBlob;2982, array)
#7 /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php(289): WANObjectCache->getWithSetCallback(string, integer, Closure$MediaWiki\Storage\SqlBlobStore::getBlob;2982, array)
#8 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionStore.php(1444): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#9 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionStore.php(1653): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#10 [internal function]: Closure$MediaWiki\Revision\RevisionStore::constructSlotRecords(MediaWiki\Revision\SlotRecord)
#11 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/SlotRecord.php(307): call_user_func(Closure$MediaWiki\Revision\RevisionStore::constructSlotRecords;2978, MediaWiki\Revision\SlotRecord)
#12 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#13 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RenderedRevision.php(226): MediaWiki\Revision\RevisionRecord->getContent(string, integer, NULL)
#14 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionRenderer.php(222): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#15 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionRenderer.php(151): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#16 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#3(MediaWiki\Revision\RenderedRevision, array)
#17 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RenderedRevision.php(197): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#3;3258, MediaWiki\Revision\RenderedRevision, array)
#18 /srv/mediawiki/php-1.34.0-wmf.17/includes/poolcounter/PoolWorkArticleView.php(196): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#19 /srv/mediawiki/php-1.34.0-wmf.17/includes/poolcounter/PoolCounterWork.php(125): PoolWorkArticleView->doWork()
#20 /srv/mediawiki/php-1.34.0-wmf.17/includes/page/Article.php(776): PoolCounterWork->execute()
#21 /srv/mediawiki/php-1.34.0-wmf.17/includes/actions/ViewAction.php(63): Article->view()
#22 /srv/mediawiki/php-1.34.0-wmf.17/includes/MediaWiki.php(507): ViewAction->show()
#23 /srv/mediawiki/php-1.34.0-wmf.17/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
#24 /srv/mediawiki/php-1.34.0-wmf.17/includes/MediaWiki.php(892): MediaWiki->performRequest()
#25 /srv/mediawiki/php-1.34.0-wmf.17/includes/MediaWiki.php(523): MediaWiki->main()
#26 /srv/mediawiki/php-1.34.0-wmf.17/index.php(42): MediaWiki->run()
#27 /srv/mediawiki/w/index.php(3): include(string)
#28 {main}
ArielGlenn added a comment.Aug 9 2019, 2:26 PM

The entry in the text row points to a non-existent blob in a cluster.

wikiadmin@10.64.48.34(zhwiki)> select * from text where old_id = 9375723;
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| old_id  | old_namespace | old_title | old_text         | old_comment | old_user | old_user_text | old_timestamp | old_minor_edit | old_flags           | inverse_timestamp |
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
| 9375723 |             0 |           | DB://cluster20/0 |             |        0 |               |               |              0 | utf-8,gzip,external |                   |
+---------+---------------+-----------+------------------+-------------+----------+---------------+---------------+----------------+---------------------+-------------------+
1 row in set (0.00 sec)

The id of 0 after the cluster20 address is the issue, just like other entries on this ticket.

Marostegui subscribed.Aug 20 2019, 12:54 PM

This happened again today :https://logstash.wikimedia.org/app/kibana#/dashboard/Fatal-Monitor?_g=h@8446ffb&_a=h@40bd263)
Example

/srv/mediawiki/multiversion/MWScript.php dumpBackup.php --wiki=nlwiktionary --full --stub --report=1000 --output=file:/mnt/dumpsdata/xmldatadumps/temp/n/nlwiktionary/nlwiktionary-20190820-stub-meta-history.xml.gz.inprog_tmp --output=file:/mnt/dumpsdata/xmldatadumps/temp/n/nlwiktionary/nlwiktionary-20190820-stub-meta-current.xml.gz.inprog_tmp --filter=latest --output=file:/mnt/dumpsdata/xmldatadumps/temp/n/nlwiktionary/nlwiktionary-20190820-stub-articles.xml.gz.inprog_tmp --filter=latest --filter=notalk --filter=namespace:!NS_USER --skip-header --start=56300 --skip-footer --end 61300

PHP Warning: MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row 4041. [Called from MediaWiki\Storage\SqlBlobStore::fetchBlob in /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php at line 361]

 MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /srv/mediawiki/php-1.34.0-wmf.17/includes/debug/MWDebug.php(309): trigger_error(string, integer)
#2 /srv/mediawiki/php-1.34.0-wmf.17/includes/debug/MWDebug.php(164): MWDebug::sendMessage(string, array, string, integer)
#3 /srv/mediawiki/php-1.34.0-wmf.17/includes/GlobalFunctions.php(1078): MWDebug::warning(string, integer, integer, string)
#4 /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php(361): wfLogWarning(string)
#5 /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php(286): MediaWiki\Storage\SqlBlobStore->fetchBlob(string, integer)
#6 /srv/mediawiki/php-1.34.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1416): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL)
#7 /srv/mediawiki/php-1.34.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1271): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array)
#8 /srv/mediawiki/php-1.34.0-wmf.17/includes/Storage/SqlBlobStore.php(288): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#9 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionStore.php(1444): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#10 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionStore.php(1653): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#11 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#12 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/SlotRecord.php(307): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#13 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/SlotRecord.php(551): MediaWiki\Revision\SlotRecord->getContent()
#14 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionSlots.php(200): MediaWiki\Revision\SlotRecord->getSha1()
#15 [internal function]: MediaWiki\Revision\RevisionSlots->MediaWiki\Revision\{closure}(NULL, MediaWiki\Revision\SlotRecord)
#16 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionSlots.php(202): array_reduce(array, Closure, NULL)
#17 /srv/mediawiki/php-1.34.0-wmf.17/includes/Revision/RevisionStoreRecord.php(174): MediaWiki\Revision\RevisionSlots->computeSha1()
#18 /srv/mediawiki/php-1.34.0-wmf.17/includes/export/XmlDumpWriter.php(309): MediaWiki\Revision\RevisionStoreRecord->getSha1()
#19 /srv/mediawiki/php-1.34.0-wmf.17/includes/export/XmlDumpWriter.php(390): XmlDumpWriter->invokeLenient(MediaWiki\Revision\RevisionStoreRecord, string, array, string)
#20 /srv/mediawiki/php-1.34.0-wmf.17/includes/export/WikiExporter.php(531): XmlDumpWriter->writeRevision(stdClass, array)
#21 /srv/mediawiki/php-1.34.0-wmf.17/includes/export/WikiExporter.php(474): WikiExporter->outputPageStreamBatch(Wikimedia\Rdbms\ResultWrapper, stdClass)
#22 /srv/mediawiki/php-1.34.0-wmf.17/includes/export/WikiExporter.php(288): WikiExporter->dumpPages(string, boolean)
#23 /srv/mediawiki/php-1.34.0-wmf.17/includes/export/WikiExporter.php(173): WikiExporter->dumpFrom(string, boolean)
#24 /srv/mediawiki/php-1.34.0-wmf.17/maintenance/includes/BackupDumper.php(289): WikiExporter->pagesByRange(integer, integer, boolean)
#25 /srv/mediawiki/php-1.34.0-wmf.17/maintenance/dumpBackup.php(82): BackupDumper->dump(integer, integer)
#26 /srv/mediawiki/php-1.34.0-wmf.17/maintenance/doMaintenance.php(99): DumpBackup->execute()
#27 /srv/mediawiki/php-1.34.0-wmf.17/maintenance/dumpBackup.php(144): require_once(string)
#28 /srv/mediawiki/multiversion/MWScript.php(101): require_once(string)

And it doesn't reference any es:

root@db1078.eqiad.wmnet[nlwiktionary]> select old_id,old_text from text where old_id=4041;
+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| old_id | old_text                                                                                                                                                                              |
+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|   4041 | %‹±^M^B1^P^Dó¯â$^B'˜^F¨à^C"^P^AÖ^E+ÞB'¬3²�>°¾bšà^LÙÎîNï¾ÀoÛ4õî5¿uä�sî‚$P^Kä)„¹^M„Væ£a^C^R3=b}•üŒJ¢ô»|ö¶ž¡tB^QÍ̃¯hr^_rÃbÖÿ^^Âm•ÚbIPk^O_                                                                  |
+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
daniel added a comment.Aug 20 2019, 2:01 PM

I suppose it's just a broken gzip archive. I didn't manage to verify since I can't find an easy way to get the raw binary value out of the database.

If we have broken data in the database, there really isn't much MediaWiki can do except log warnings. If we don't have a way to fix the data, we could instead blank it out.

Anomie added a comment.Aug 20 2019, 3:23 PM

The nlwiktionary row with old_id=4041 does indeed seem to be an invalid gzip blob.

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
WDoranWMF triaged this task as Medium priority.Sep 3 2019, 6:37 PM
WDoranWMF edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering (Needs Cleaning - Security, stability, performance, and scalability (TEC1)).
daniel moved this task from Inbox to Backlog on the Platform Team Workboards (Clinic Duty Team) board.Oct 8 2019, 9:57 AM
Krinkle mentioned this in T237570: EditPage.php: Call to a member function getModel() on null.Nov 6 2019, 7:34 PM
Krinkle added a project: MediaWiki-Core-Revision-backend.Nov 6 2019, 7:37 PM
Krinkle added a subscriber: Umherirrender.

From another task:

In T220160#5105271, @Umherirrender wrote:

The redirect page (192386) is not working:

https://hr.wikipedia.org/wiki/Franjeva%C4%8Dki_samostan_Sv._Antuna_Padovanskog_u_Koprivnici

This seems to be due to the issue described in this task, and makes this title inaccessible on our site:

RevisionAccessException from line 1442 of /srv/mediawiki/php-1.35.0-wmf.4/includes/Revision/RevisionStore.php: Failed to load data blob from tt:1677927: Bad data in text row 1677927.
jcrespo subscribed.Dec 20 2019, 7:38 AM

Spike on oswiki: https://logstash.wikimedia.org/goto/d307e0ba6dcfeb8fc748c17d3672d1ef

Marostegui added a comment.Dec 20 2019, 7:56 AM

Spike on jvwiki: https://logstash.wikimedia.org/goto/5b5b1f21a14c3f5d048c252674c75b42

ArielGlenn added a comment.Dec 20 2019, 9:32 AM

I checked a random one from oswiki which was associated witha revision with timestamp 20090309213655. The revision for that in the previous full dump has missing content and sha1 so this is not new. I guess the same will turn out to be true for the other revisions.

Krinkle moved this task from Untriaged to BlobStore on the MediaWiki-Core-Revision-backend board.Mar 10 2020, 1:32 AM
daniel moved this task from Backlog to Later on the Platform Team Workboards (Clinic Duty Team) board.Apr 14 2020, 8:59 PM
thcipriani subscribed.Jun 3 2020, 6:23 PM

Seeing this again for a different revision/page:

id: AXJ62XSusWch-KE7JvV6

message: Failed to load data blob from tt:9375723: Bad data in text row 9375723.

stacktrace
#0 /srv/mediawiki/php-1.35.0-wmf.34/includes/Revision/RevisionStore.php(1014): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#1 /srv/mediawiki/php-1.35.0-wmf.34/includes/Revision/RevisionStore.php(1247): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#2 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#3 /srv/mediawiki/php-1.35.0-wmf.34/includes/Revision/SlotRecord.php(307): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#4 /srv/mediawiki/php-1.35.0-wmf.34/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#5 /srv/mediawiki/php-1.35.0-wmf.34/includes/page/WikiPage.php(798): MediaWiki\Revision\RevisionRecord->getContent(string, integer, NULL)
#6 /srv/mediawiki/php-1.35.0-wmf.34/includes/page/WikiPage.php(1056): WikiPage->getContent()
#7 /srv/mediawiki/php-1.35.0-wmf.34/includes/page/WikiPage.php(1043): WikiPage->insertRedirect()
#8 /srv/mediawiki/php-1.35.0-wmf.34/includes/page/WikiPage.php(1122): WikiPage->getRedirectTarget()
#9 /srv/mediawiki/php-1.35.0-wmf.34/includes/MediaWiki.php(450): WikiPage->followRedirect()
#10 /srv/mediawiki/php-1.35.0-wmf.34/includes/MediaWiki.php(303): MediaWiki->initializeArticle()
#11 /srv/mediawiki/php-1.35.0-wmf.34/includes/MediaWiki.php(978): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.35.0-wmf.34/includes/MediaWiki.php(535): MediaWiki->main()
#13 /srv/mediawiki/php-1.35.0-wmf.34/index.php(47): MediaWiki->run()
#14 /srv/mediawiki/w/index.php(3): require(string)
#15 {main}
thcipriani renamed this task from Warning: MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row 5191150 to Warning: MediaWiki\Storage\SqlBlobStore::fetchBlob: Bad data in text row.Jun 3 2020, 6:23 PM
Krinkle added a subtask: T205936: Unable to view some pages due to fatal RevisionAccessException: "Failed to load data blob from tt".Jun 17 2020, 12:44 AM
daniel closed subtask T205936: Unable to view some pages due to fatal RevisionAccessException: "Failed to load data blob from tt" as Resolved.Aug 26 2020, 1:21 PM
thcipriani raised the priority of this task from Medium to High.Oct 14 2020, 6:41 PM

Happening on hrwiki as of today (2020-10-14) with wmf.11: Failed to load data blob from tt:1677927: Bad data in text row 1677927. Use findBadBlobs.php to remedy.. If this problem persist, use the findBadBlobs maintenance script to investigate the issue and mark bad blobs.

Are these reports helpful? Should this be an exception if this means we need to run findBadBlobs? Should we just run that job on a schedule?

ArielGlenn added a comment.Oct 14 2020, 6:50 PM

I'd like eventually to run it across all wikis and get a sense of the bad timeframes. Just my 2 cents.

daniel added a comment.EditedOct 15 2020, 8:52 AM
In T203075#6543812, @thcipriani wrote:

Are these reports helpful? Should this be an exception if this means we need to run findBadBlobs? Should we just run that job on a schedule?

We should not run this script automatically in "mark" mode. That would mean we would ignore data corruption, and potentially make it worse. The cause should always be investigated. The script is potentially destructive, it basically tells MediaWiki to give up on the content of the revision. We should only do that if there is no chance of restoring the content that was lost.

As an example, if one of the External Store hosts has a temporary problem, we would be seeing this error. If we ran the script to mark the revisions, we'd permanently mark them as broken, instead of just letting things recover once the ES box has been taken out of rotation or got repaired.

We could run the script over all wikis to find corrupt revisions, but the script is slow. Right now, it can only be run against a known set of revisions or a limited time range.

In any case, MediaWiki should always report an error when reading a revision fails due to data corruption. Ignoring such incidences would be dangerous. The fact that we see this error again and again is that it's a symptom that can have a variety of causes.

daniel added a comment.EditedOct 15 2020, 8:58 AM

Given that this error may be a symptom of a large variety of problems, from temporary network glitches to deep rooted logic errors, it probably doesn't make sense to treat all occurrences as instances of the same issue. Every time we see this, we need to check *why* the data can't be loaded. Only after we know that can we decide how the problem can be addressed. This means every instance of this should be tracked separately. Perhaps there should be a note to this effect in the description of this ticket. And perhaps this should become a tracking ticket, with individual occurrences as subtasks.

Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:01 PM
ArielGlenn added a comment.Oct 26 2020, 4:54 PM

See T265989 where I have collected a bunch of bad revisions with timestamps across all the wikis. This may let us make some headway.

Krinkle closed this task as Resolved.Jan 27 2021, 8:08 PM
Krinkle claimed this task.

Marking as resolved because I'm unable to reproduce new entries for this based on the history of the dewiki "Ahmadiyya" article. I've browsed quite far bar and none of them yielded warnings. Per Daniel and Ariel above, this is too broad a category of issues to keep a single task about, and we in fact alreayd have dozens of tasks for this, so I'll just close this.

Note that I'm not assuming this issue to have been intermittent, rather we did a sweep recently to repair and/or mark various known empty/absent text blobs as such, thus no longer causing run-time warnings.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits