@Helpful_Raccoon: In future, please copy and paste the full error messages, as it allows the text to be selected/searched.

I am getting this error on at least one other archive page: https://en.wikipedia.org/wiki/Talk:Donald_Trump/Archive_107.
[dca5c09c-1703-4a35-b1f2-db3cdce4d179] 2025-05-03 08:24:07: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"

I was seeing this on https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(policy). At least for that case, I tracked it down to something being broken with https://en.wikipedia.org/wiki/Template:Purple.

Checking various other pages transcluding Template:Purple, some are showing the error currently, while others seem to work from cache for the moment but attempting to preview an edit breaks. Attempting to edit or delete Template:Purple itself also fails with the same message about a RevisionAccessException.

[13a24de6-3fd1-4757-ad2d-d04126119d13] /wiki/Template:Purple   MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274359757. Use findBadBlobs.php to remedy. for revision 488705891. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.

https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-deploy-1-7.0.0-1-2025.05.03?id=MPsWl5YBuXzFNByTABUc

See also T387683, which may or may not be the same error.

Tagging DBA for help, based on the last time this happened (T386162).

Relevant Logstash query: https://logstash.wikimedia.org/goto/60b13e81a8e5c01196330f64399c4908

There is a very slow trickle of 1-10 errors per day (affecting different revisions on different wiki), then a big spike of the errors from Template:Purple on enwiki, with the first one on May 2, 2025 @ 23:26:41.784.

This was caused by T183490. I ran "stage 2"[1] of that migration on s1 (aka enwiki), i.e. ~5.9 million rows where deleted from text table which were expect to no longer be needed.

But for some reason I cannot explain, 431 of those rows are actually still in use.

[1] - https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaMaintenance/+/refs/heads/master/migrateESRefToContentTableStage2.php

We looked at the binlogs and it appears that the current revision of the purple template is corrupted, so I went ahead marked that as bad and created a new revision with the correct content.

zabe@mwmaint1002:~$ mwscript findBadBlobs.php enwiki --revisions 488705891 --mark "T393237"
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
Scanning 1 ids
        ! Found bad blob on revision 488705891 from 20120422202619 (main slot): content_id=444499441, address=<tt:274359757>, error='Unable to fetch blob at tt:274359757. Use findBadBlobs.php to remedy.', type='MediaWiki\Storage\BlobAccessException'. ID:      488705891
        Changed address to <bad:tt%3A274359757?reason=T393237&error=Unable+to+fetch+blob+at+tt%3A274359757.+Use+findBadBlobs.php+to+remedy.>
        - Scanned a batch of 1 revisions
Marked 1 bad revisions.
zabe@mwmaint1002:~$ mwscript shell.php enwiki
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
Psy Shell v0.12.7 (PHP 7.4.33 — cli) by Justin Hileman
> $cache = \MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
= Wikimedia\ObjectCache\WANObjectCache {#902}

> $key = $cache->makeKey('revision-slots', '', 14350025, 488705891);
= "enwiki:revision-slots::14350025:488705891"

> $cache->delete( $key );
= true

> ^D

   INFO  Ctrl+D.

zabe@mwmaint1002:~$

Note, viewing the old version still results in a mediawiki error

https://en.wikipedia.org/w/index.php?title=Template:Purple&diff=prev&oldid=488705891

In T393237#10789204, @Xaosflux wrote:

Note, viewing the old version still results in a mediawiki error

https://en.wikipedia.org/w/index.php?title=Template:Purple&diff=prev&oldid=488705891

Also marked the two previous revisions as bad. Please note that there are multiple caching layers, so it will take a bit for that to be visible.

zabe@mwmaint1002:~$ mwscript findBadBlobs.php enwiki --revisions 488567353,276113208 --mark "T393237"
DEPRECATION WARNING: Maintenance scripts are moving to Kubernetes. See
https://wikitech.wikimedia.org/wiki/Maintenance_scripts for the new process.
Maintenance hosts will be going away; please submit feedback promptly if
maintenance scripts on Kubernetes don't work for you. (T341553)
Scanning 2 ids
        ! Found bad blob on revision 276113208 from 20090309211539 (main slot): content_id=247612691, address=<tt:274359757>, error='Unable to fetch blob at tt:274359757. Use findBadBlobs.php to remedy.', type='MediaWiki\Storage\BlobAccessException'. ID:      276113208
        Changed address to <bad:tt%3A274359757?reason=T393237&error=Unable+to+fetch+blob+at+tt%3A274359757.+Use+findBadBlobs.php+to+remedy.>
        ! Found bad blob on revision 488567353 from 20120421230203 (main slot): content_id=444370439, address=<tt:274359757>, error='Unable to fetch blob at tt:274359757. Use findBadBlobs.php to remedy.', type='MediaWiki\Storage\BlobAccessException'. ID:      488567353
        Changed address to <bad:tt%3A274359757?reason=T393237&error=Unable+to+fetch+blob+at+tt%3A274359757.+Use+findBadBlobs.php+to+remedy.>
        - Scanned a batch of 2 revisions
Marked 2 bad revisions.
zabe@mwmaint1002:~$

One of the bad revisions appears to be cleared (giving badrevision result), however this is still failing with error:

https://en.wikipedia.org/w/index.php?title=Template:Purple&oldid=488567353

In T393237#10789308, @Xaosflux wrote:

One of the bad revisions appears to be cleared (giving badrevision result), however this is still failing with error:

https://en.wikipedia.org/w/index.php?title=Template:Purple&oldid=488567353

Note, this is now marked as a bad version

Out of nowhere we are getting the same error for a different page. The address is part of the 431 listed above. @Ladsgroup could you look if the binlogs do contain something for this one?

• service.version: 1.44.0-wmf.27
• trace.id: f52ca013-0dbc-4d9c-ad84-b2868a676b4c
• Find trace.id in Logstash

Failed to load data blob from Unable to fetch blob at tt:274392442. Use findBadBlobs.php to remedy. for revision 819689534. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.

The same issue:

### DELETE FROM `enwiki`.`text`
### WHERE
###   @1=792737651 /* INT meta=0 nullable=0 is_null=0 */
###   @2='DB://cluster25/140977019' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
###   @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */
--
### DELETE FROM `enwiki`.`text`
### WHERE
###   @1=274392442 /* INT meta=0 nullable=0 is_null=0 */
--
### DELETE FROM `enwiki`.`text`
### WHERE
###   @1=724987058 /* INT meta=0 nullable=0 is_null=0 */
###   @2='DB://cluster25/106413391' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
###   @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */
--

It's also clear that it's not a log rotation at middle of transaction issue.

Mentioned in SAL (#wikimedia-operations) [2025-05-07T15:49:21Z] <zabe> zabe@mwmaint1002:~$ mwscript findBadBlobs.php enwiki --revisions 276146284,819689534,1289169661 --mark "T393237"

I suggest running findBadBlobs on all of 400 revs and call it a day.

How was it possible that 400 revisions were lost here? Why were backups of the text table not taken that could be used to recover from this mistake? Just accepting that the content of 400 revisions is now gone forever seems remarkably cavalier to me.

In T393237#10807988, @Pppery wrote:

How was it possible that 400 revisions were lost here? Why were backups of the text table not taken that could be used to recover from this mistake? Just accepting that the content of 400 revisions is now gone forever seems remarkably cavalier to me.

1- 400 in 1,280,000,000 edits is nothing.
2- We confirmed they were corrupted before the migration. Mistake didn't happen in this case. Those edits all are old and from a time we had way less protections in production.

In T393237#10807998, @Ladsgroup wrote:

In T393237#10807988, @Pppery wrote:

2- We confirmed they were corrupted before the migration. Mistake didn't happen in this case. Those edits all are old and from a time we had way less protections in production.

The revisions were old, but they certainly existed until recently. Template:Purple (whose latest revision was affected) was accessible until recently (https://web.archive.org/web/20250214182330/https://en.wikipedia.org/wiki/Template:Purple), and so were pages using the template (e.g. https://web.archive.org/web/20220607205254/https://en.wikipedia.org/wiki/Declension). This looks to me like we lost the data recently, not so long ago that we can't do anything about it.

In T393237#10796606, @Zabe wrote:

Out of nowhere we are getting the same error for a different page. The address is part of the 431 listed above.

Presumably this error was masked at first due to the caching layers mentioned in T393237#10789205?

In T393237#10788958, @Zabe wrote:

This was caused by T183490. I ran "stage 2"[1] of that migration on s1 (aka enwiki), i.e. ~5.9 million rows where deleted from text table which were expect to no longer be needed.

But for some reason I cannot explain, 431 of those rows are actually still in use.

The two revisions that caused widespread issues, and – I just checked – the majority of the affected revisions, are revisions that are followed by page protections or moves, which create dummy revisions with identical text. This is probably a hint. Maybe the text rows could be associated with multiple revision rows, but the script assumes a one-to-one relationship and deleted too much?

Sample:

They also all were originally created in a narrow time range, between 9 and 10 March 2009. According to Server Admin Log archives (2008 Oct - 2009 Jun → March 9-10), there was some trouble with the revision external storage on enwiki on those exact days:

March 10

• 00:52 brion: domas makes it all better. yay domas!
• 00:50 brion: cluster21 master (srv161) in read-only; we're having write failure problems on multiple wikis
• 00:48 brion: putting ES back into service on enwiki; srv160/cluster20 master has been fixed. slaves still running it, but this is safe for read/write since reads will fall back to master

March 9

• 20:15ish-20:30ish Brion:
  • some (unspecified) blob tables in ES broken due to MAX_ROWS=1m. writes broken on enwiki. domas is rebuilding tables
  • disabling wgDefaultExternalStore on enwiki temporarily as hack measure. last text_id was 274347580 before
  • now actually disabling read-only too

So presumably these revisions were written in a funny way, or maybe already processed with some other maintenance scripts that modified them in a funny way.

(Reminder to please log your one-off database-altering scripts… You never know when this will answer someone's question 15 years later)

In T393237#10808846, @matmarex wrote:

In T393237#10807998, @Ladsgroup wrote:

In T393237#10807988, @Pppery wrote:

2- We confirmed they were corrupted before the migration. Mistake didn't happen in this case. Those edits all are old and from a time we had way less protections in production.

The revisions were old, but they certainly existed until recently. Template:Purple (whose latest revision was affected) was accessible until recently (https://web.archive.org/web/20250214182330/https://en.wikipedia.org/wiki/Template:Purple), and so were pages using the template (e.g. https://web.archive.org/web/20220607205254/https://en.wikipedia.org/wiki/Declension). This looks to me like we lost the data recently, not so long ago that we can't do anything about it.

In T393237#10796606, @Zabe wrote:

Out of nowhere we are getting the same error for a different page. The address is part of the 431 listed above.

Presumably this error was masked at first due to the caching layers mentioned in T393237#10789205?

For the second occurrence mentioned above the content of the revision was indeed cached in the SqlBlobStore cache.

In T393237#10788958, @Zabe wrote:

This was caused by T183490. I ran "stage 2"[1] of that migration on s1 (aka enwiki), i.e. ~5.9 million rows where deleted from text table which were expect to no longer be needed.

But for some reason I cannot explain, 431 of those rows are actually still in use.

The two revisions that caused widespread issues, and – I just checked – the majority of the affected revisions, are revisions that are followed by page protections or moves, which create dummy revisions with identical text. This is probably a hint. Maybe the text rows could be associated with multiple revision rows, but the script assumes a one-to-one relationship and deleted too much?

T393237.sql7 KBDownload

Sample:

rev_id	content_address	comment_text
505952912	tt:274350127	Anaxial moved page [[Talk:Mexican Cottontail]] to [[Talk:Mexican cottontail]]: Per [[WP:FNAME]]
282162613	tt:274350152	moved [[Talk:The Rockettes]] to [[Talk:The Rockettes (dance company)]]: with only the definite article making the difference, there are two similarly-named groups, a dance company and a synchro skating team
316302510	tt:274350153	moved [[Talk:Amphitryon (1935 film)]] to [[Talk:Amphitryon (film)]]
276186194	tt:274350364	Protected Tennessee: [[WP:MOVP|Page-move vandalism]]: target ([move=sysop] (indefinite))
651596623	tt:274350442	Corkythehornetfan moved page [[Template talk:Truman State Bulldogs football coach navbox]] to [[Template talk:Truman Bulldogs football coach navbox]]: Truman does not use 'State' in their name

Yes indeed all of these former text table rows do have multiple content rows references to them. This is something that should not happen (dummy revisions should reuse the former content row), but we did realise this at some point (T373668). So we used a bloom filter to gather a list of text table rows which are referenced multiple times from the content table (you can find that list at mwmaint1002.eqiad.wmnet:/home/zabe/text_table_cleanup/enwiki). You will notice that the above 431 listed rows are a subset of this list.

In general what happened is that I ran migrateESRefToContentTable.php on enwiki. We did pass the former list of duplicate references to it, and it would skip deleting those for now. In line 192 of that script, it writes the es address of every content row updated to a file (for enwiki you can currently find this file at stat1009.eqiad.wmnet:/home/zabe/enwikitest/enwiki) regardless of whether the corresponding text row is being deleted or not. Sometimes this did not work (i.e. T392834), but it should cover most of it (the file is 69GB and contains 1.209.020.606 lines).

So for those 431 rows what should have happened is that on the first occurrence in the content table, the corresponding external storage address should have been written to the content_address field and the mapping from the text table row to the external storage address should have been written to the dump. But both of these failed. Also both of these failed again on the second (or third) occurrence of this text table row in the content table.

In the end I ran migrateESRefToContentTableStage2.php assuming all references to the text table, where it was possible, were migrated (this assumption was mostly based on the fact that the content table only contained around 66.000 remaining references, while the number of duped content rows was around 5,8 million, so that those must have been migrated; it turned out 431 were not). These deletions are the ones we looked at at the binlogs above.

So in the end the current this was certainly caused by me running the stage 2 script, the 431 named rows certainly were semi corrupted in some way which somehow allowed reading its content but touching them did cause issues.

There are 3 main indicators for this, 1. the script should not at all randomly skip the content table rows referencing these text table rows without any reason, especially since the script did it at least two times for each of those rows, 2. the binlogs did not, as they usually do, showed the previous content of the row when they were deleted (T393237#10799542) and 3. the sal entries you found which show that there were external storage saving issues at the time of the creation of these rows.

Thanks for the additional details. I'm still struggling to understand how the data could be available for reading by MediaWiki for all these years, but could not be updated by the maintenance script that should have migrated it, but I'll take your word for it. I guess it's too late to hope to review the script logs?

The data seems to also be available in the XML dumps – I checked just one page, but enwiki-20250501-pages-meta-history15.xml-p14324603p14439634 has all of the revisions of Template:Purple, including the missing ones. If someone really wanted, they could probably restore it from there.

In T393237#10809023, @matmarex wrote:

Thanks for the additional details. I'm still struggling to understand how the data could be available for reading by MediaWiki for all these years, but could not be updated by the maintenance script that should have migrated it, but I'll take your word for it. I guess it's too late to hope to review the script logs?

Tbh I also do not understand how that could have been the case. Since the script ran over a period of 7 months and was restarted multiple times the script log is no longer there.

The data seems to also be available in the XML dumps – I checked just one page, but enwiki-20250501-pages-meta-history15.xml-p14324603p14439634 has all of the revisions of Template:Purple, including the missing ones. If someone really wanted, they could probably restore it from there.

A few recent hits from train log triage:

[280b2715-2781-4d66-a00e-ba6417723306] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2   MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274365094. Use findBadBlobs.php to remedy. for revision 832352351. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
[da0c8b3f-7fbe-4c07-9cd2-f13504e9c49a] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2   MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274360148. Use findBadBlobs.php to remedy. for revision 693354709. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
[6a429a86-2c58-4d75-a201-e04c2eaf457b] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2   MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274395852. Use findBadBlobs.php to remedy. for revision 657587061. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.
[d4a5cc69-5210-42e0-b5b3-fbbc5a3ec5ec] /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json&formatversion=2   MediaWiki\Revision\RevisionAccessException: Failed to load data blob from Unable to fetch blob at tt:274372796. Use findBadBlobs.php to remedy. for revision 561641691. If this problem persists, use the findBadBlobs maintenance script to investigate the issue and mark the bad blobs.

Can someone run findBadBlobs?

Mentioned in SAL (#wikimedia-operations) [2026-04-10T00:29:27Z] <zabe> marked 425 content rows as bad # T393237

Marked the following revisions as bad. Please note that since revision may share the same content row, and per above analysis this should be the case for every single one of these, this also effects at least another 425 revisions.

I will double check in the next few days whether there are any revisions left that need to be marked as bad.

276101121,276101318,276101514,276101519,276101568,276101650,276101759,276101775,276101817,276101854,276102034,276102062,276102161,276102190,276102732,276102750,276102938,276103027,276103326,276103504,276103529,276103530,276103741,276103819,276104163,276104327,276104377,276104631,276104807,276104862,276104933,276105055,276105106,276105322,276105484,276105515,276105621,276105921,276105932,276105982,276106006,276106027,276106099,276106243,276106276,276106298,276106378,276106520,276106638,276106693,276106856,276106944,276107045,276107104,276107202,276107253,276107301,276107344,276107382,276107487,276107582,276107616,276107675,276107899,276107930,276108121,276108124,276108195,276108196,276108197,276108272,276108599,276108901,276108922,276108990,276109009,276109031,276109148,276109365,276109471,276109519,276110092,276110215,276110313,276110447,276110689,276110723,276110786,276110924,276111155,276111160,276111180,276111344,276111393,276111555,276111679,276111693,276111848,276111979,276112167,276112172,276112245,276112260,276112366,276112387,276112435,276112485,276112722,276112773,276113050,276113279,276113605,276113678,276113765,276113965,276114262,276114306,276114465,276114627,276114885,276115329,276115640,276115721,276115811,276115886,276116114,276116380,276116520,276116625,276116680,276116692,276116757,276116853,276116962,276117086,276117309,276117331,276117552,276117641,276117946,276117971,276118004,276118099,276118183,276118400,276118403,276118600,276118602,276118670,276118785,276118845,276119062,276119114,276119146,276119167,276119232,276119247,276119516,276119545,276120019,276120130,276120281,276120422,276120489,276120987,276121108,276121289,276121382,276121424,276121507,276121646,276121669,276121713,276122006,276122016,276122089,276122614,276123108,276123152,276123190,276123823,276123987,276124093,276124494,276124503,276124523,276124920,276125049,276125223,276125462,276125926,276126149,276126330,276126366,276126464,276126934,276127591,276127617,276127805,276127887,276127988,276128106,276128167,276128199,276128700,276128815,276128996,276129214,276129349,276129919,276130033,276130591,276130872,276130932,276131000,276131184,276131425,276131563,276131651,276131672,276131699,276131783,276132164,276132265,276132662,276132859,276133133,276133309,276133352,276133448,276133482,276133607,276133811,276133824,276134040,276134292,276134408,276134486,276134753,276134945,276134964,276135638,276135707,276135858,276136194,276136464,276136467,276136776,276136822,276136916,276138024,276138092,276138283,276138714,276138791,276139376,276139463,276139611,276139617,276139908,276140000,276140136,276140298,276140303,276140578,276140592,276140631,276140753,276141166,276141237,276141339,276141569,276141705,276141752,276141792,276141812,276141857,276142085,276142088,276142218,276142355,276142486,276142806,276143522,276143605,276144152,276144180,276144223,276144424,276144479,276144508,276144681,276144723,276144772,276144787,276144801,276144970,276145170,276145285,276145494,276145729,276145881,276146024,276146123,276146199,276146241,276146349,276146418,276146425,276146524,276147068,276147085,276147391,276147662,276147802,276147881,276147986,276148067,276148077,276148127,276148385,276148429,276148690,276148813,276149182,276149403,276149501,276149542,276149715,276149723,276149774,276149880,276149944,276150136,276150199,276150222,276150267,276150347,276150477,276150565,276150632,276150655,276150675,276150678,276150806,276150855,276150863,276151106,276151622,276152220,276152331,276152513,276152546,276152631,276152830,276152990,276153008,276153076,276153108,276153126,276153189,276153265,276153328,276153733,276153762,276154225,276154232,276154449,276154560,276154781,276154904,276154938,276155090,276155722,276155729,276155889,276155974,276156157,276156437,276157140,276157521,276157576,276158010,276158090,276158192,276158406,276158790,276158930,276159008,276159156,276159251,276159296,276159297,276159666,276159945,276159990,276160181,276160205,276160440,276160708,276160771,276160820,276160933,276162488,276162674,276163370,276163374,276163805,276164025,276164809,276166874,276177631,276178908,276179429,276179768,276180243,276184044,276184841,276185772,276185998,276186194,276186958,276187976,276198954,276198958