We get the same Error-Message if we try to add a Template to an article.

https://de.wiktionary.org/w/index.php?title=Prompt&action=edit

When we add ", {{Pl.}}" after the text ":Prompt" (see below)

{{Worttrennung}}
:Prompt

can be saved

{{Worttrennung}}
:Prompt, {{Pl.}}

cannot be saved. We get the message
[7101a73e-7540-40d7-ac03-f9e1601a0c71] 2025-02-12 10:20:13: Fataler Ausnahmefehler des Typs „MediaWiki\Revision\RevisionAccessException“

The page https://de.wiktionary.org/wiki/Wiktionary:Autorenportal could not be read any more.

I'm going through binlogs to see what was the values of the rows.

I ran select count(*) from content where content_address like 'tt%'; on all group1 wikis and for dewiktionary it states 9649703. So apparently the first stage crashed at somepoint and it just continued with the next wiki in that shard.

Can you tell me when the deletions have happened? I went through 20 binlogs and still can't find the delete queries.

Around 1 AM today.

I can't find it in the binlogs. I find deletions on other tables of dewiktionary but not text:

root@db2186:/srv/sqldata.s3# sudo mysqlbinlog -vv --base64-output=DECODE-ROWS db2186-bin.001558 | grep -i -C10 "DELETE FROM \`dewiktionary\`\.\`text" | head --lines 50
root@db2186:/srv/sqldata.s3# ls -l | grep -i db2186-bin.001558
-rw-rw---- 1 mysql mysql 1048588175 Feb 12 02:24 db2186-bin.001558
root@db2186:/srv/sqldata.s3# sudo mysqlbinlog -vv --base64-output=DECODE-ROWS db2186-bin.001557 | grep -i -C10 "DELETE FROM \`dewiktionary\`\.\`text" | head --lines 50
root@db2186:/srv/sqldata.s3#

Siiiiiiigh, not finding the deletion logs was my fault. I'm checking sanitarium to avoid touching live production while forgetting that text table gets filtered there and doesn't exist. Let me check another production host.

I found the deletions and they are happening a lot. Please give the list of tt ids so I can tell you what was the pre-deletion value so we can add them to content table.

This is the first one:

### DELETE FROM `dewiktionary`.`text`
### WHERE
###   @1=1293124 /* INT meta=0 nullable=0 is_null=0 */
###   @2='DB://cluster22/318542' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
###   @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */

UPDATE content set content_address = 'es:DB://rc1/8309/0?flags=utf-8' where content_id in (752811, 1161934);

UPDATE content set content_address = 'es:DB://cluster22/318542?flags=utf-8,gzip' where content_id in (1233172, 3988976);

Does this look correct to you @Zabe ?

Change #1119202 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] logspam.pl:

https://gerrit.wikimedia.org/r/1119202

In T386162#10544623, @Ladsgroup wrote:

I found the deletions and they are happening a lot. Please give the list of tt ids so I can tell you what was the pre-deletion value so we can add them to content table.

All are messed up? In that case, we should use recovery from backups then.

In T386162#10544941, @Ladsgroup wrote:

UPDATE content set content_address = 'es:DB://rc1/8309/0?flags=utf-8' where content_id in (752811, 1161934);

UPDATE content set content_address = 'es:DB://cluster22/318542?flags=utf-8,gzip' where content_id in (1233172, 3988976);

Does this look correct to you @Zabe ?

Yes.

We do get the correct content for the Pl. template:

> $es->fetchFromURL('DB://rc1/8309/0');
= """
  <span style="font-size:95%;">Plural:</span><noinclude>\n
  [[Kategorie:Wiktionary:Vorlage|Pl.]]</noinclude>
  """

>

And some gziped bla from the other one:

> $a = $es->fetchFromURL('DB://cluster22/318542?flags=utf-8,gzip');
= b"eÄ1\x0E┬0\fE¸£┬c+qàî *ûN0ó@M1ñIÕ$\x08æµ6▄äï\x11$Jæ║xx ┘▀RBÅýéjÉ\x11èì5ùð*OÍöP─©E>\'6T^iBC¯xN®\x04)Eî\x7F\f>>¼4Ì│,åØe´╝Û:¨Ý\x10├Ê4┴┤2ùëíVýÚA=ȳzzd‗a4Ø═±Dí¬÷dÃîfÖ\x0E┤ðHúÇs┴‗o\x1D9\x1F_ô>╣[■V¬<1Ñ7"

> gzinflate( $a )
= """
  == persuadere (Konjugation) ({{Verbkonjugation|Italienisch}}) ==\n
  {{Italienisch Verb Flexion\n
  |Wortstamm=persuad\n
  |Endung=ere\n
  |Partizip Präteritum=persuaso\n
  |Präteritum II_io=persuasi\n
  |Präteritum II_lui,lei=persuase\n
  |Präteritum II_loro=persuasero\n
  |Hilfsverb=avere}}
  """

>

So the flags also do look good to me.

Applied. That should fix some logspam.

In T386162#10546411, @Ladsgroup wrote:

All are messed up? In that case, we should use recovery from backups then.

Not all, but like 95%+.

For most of them we will barely notice since they represent some ancient revisions which are never looked at.

We do mostly notice the handful of ones which are the current revision of a highly used template.

In T386162#10546431, @Ladsgroup wrote:

Applied. That should fix some logspam.

I deployed https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1118888 last night in preparation to any fix. But since it isn't live for 24 hours yet, it can still take a few hours for this to take effect.

Could you please fix {{Hauptseite/Interessante Einträge}} in advance, like you did with {{Pl.}} earlier?

It looks like {{Hauptseite/Interessante Einträge}} is responsible for the fact that the main page of the German Wiktionary can no longer be displayed.

Based on binlog, I ran UPDATE content set content_address = 'es:DB://cluster24/616217?flags=utf-8,gzip' where content_id in (3798702, 5099126); which should fix tt:3891943 case that was causing a lot of logspam. Let me see how it unfolds.

The logspam should be cut to half. @Zabe if you know any important text addresses used in highly used templates, I can take care of it manually now, otherwise, I will build an automation for it tomorrow.

@Ladsgroup: Thank you very much. This has helped and the main page is working again.

Unrelated, but:

If this problem persist, use the findBadBlobs maintenance script to investigate the issue and mark bad blobs

is incorrect English grammar. It should be "if this problem persists ..."

Change #1119244 had a related patch set uploaded (by Novem Linguae; author: Novem Linguae):

[mediawiki/core@master] RevisionStore: improve grammar of error message

https://gerrit.wikimedia.org/r/1119244

Change #1119244 merged by jenkins-bot:

[mediawiki/core@master] RevisionStore: improve grammar of error message

https://gerrit.wikimedia.org/r/1119244

Building some automation right now

I extracted all the deletion binlogs, now cross referencing them with the list to shrink it down.

Writing the script is finished now. It's just querying the database to build the full mapping of old id => content address. The script produces this:

UPDATE /* tt:479972 */ content set content_address = 'es:DB://cluster9/38431?flags=utf-8,gzip' where content_id = 598259;
UPDATE /* tt:422904 */ content set content_address = 'es:DB://cluster9/9941?flags=utf-8,gzip' where content_id = 406285;
UPDATE /* tt:422904 */ content set content_address = 'es:DB://cluster9/9941?flags=utf-8,gzip' where content_id = 598261;
...

Does the write query look correct (in the sense of structure, the value is automatically generated and should be fine) @Zabe ?

In T386162#10550344, @Ladsgroup wrote:

Writing the script is finished now. It's just querying the database to build the full mapping of old id => content address.

It's only 7% done after an hour or so. It'll take a while to finish.

In T386162#10550344, @Ladsgroup wrote:

Writing the script is finished now. It's just querying the database to build the full mapping of old id => content address. The script produces this:

UPDATE /* tt:479972 */ content set content_address = 'es:DB://cluster9/38431?flags=utf-8,gzip' where content_id = 598259;
UPDATE /* tt:422904 */ content set content_address = 'es:DB://cluster9/9941?flags=utf-8,gzip' where content_id = 406285;
UPDATE /* tt:422904 */ content set content_address = 'es:DB://cluster9/9941?flags=utf-8,gzip' where content_id = 598261;
...

Does the write query look correct (in the sense of structure, the value is automatically generated and should be fine) @Zabe ?

LGTM

Awesome. I ran the first 800 that are mapped on master, so nothing exploded. If all goes well, I run it on all mapping once it's done building it.

Running the update queries

So here is a detailed explanation of what I did. For documentation purposes.

In dbstore1007 where it uses RBR. I ran this:

root@dbstore1007:/srv/sqldata.s3# ls dbstore1007-bin.00134* | xargs -I{} sh -c 'mysqlbinlog -vv --base64-output=DECODE-ROWS {} | grep -i -A4 "DELETE FROM \`dewiktionary\`.\`text"' >> /home/ladsgroup/deletions_T386162

(for multiple set of binlogs)
The output was a 1.3GB file of stuff like this:

...
###   @2='DB://cluster25/1236362' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
###   @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */
--
### DELETE FROM `dewiktionary`.`text`
### WHERE
###   @1=5114966 /* INT meta=0 nullable=0 is_null=0 */
###   @2='DB://cluster25/1236363' /* MEDIUMBLOB/MEDIUMTEXT meta=3 nullable=0 is_null=0 */
###   @3='utf-8,gzip,external' /* TINYBLOB/TINYTEXT meta=1 nullable=0 is_null=0 */
--
### DELETE FROM `dewiktionary`.`text`
### WHERE
...

then I wrote a script to query content table to find content ids using the text table ids (P73476). Ran it from cumin (to dbstore1007). It produced a json file like this:

{..."tt:964928": [917342, 3991710], "tt:587995": [560955, 3991711], "tt:1075095": [1023292, 3991712], "tt:964941": [917355, 3991713], "tt:1075102": [1023299, 3991714], "tt:964943": [917357, 3991715],..}

The next step was to combine these two files to produce the update queries. It was done via this python script (P73477). The output was something like:

UPDATE /* tt:10010118 */ content set content_address = 'es:DB://cluster29/66362?flags=utf-8,gzip' where content_id = 9997795; select sleep(0.1);
UPDATE /* tt:10011426 */ content set content_address = 'es:DB://cluster28/66705?flags=utf-8,gzip' where content_id = 9999097; select sleep(0.1);
UPDATE /* tt:10011876 */ content set content_address = 'es:DB://cluster28/66935?flags=utf-8,gzip' where content_id = 9999547; select sleep(0.1);

I spot checked them in multiple places to make we don't royally mess things up because of one-off error. So far it looked good.

I pushed the file into mwmaint2002 and ran it via sql.php. It's running but it'll take around an hour to push all 49K updates (most text table entries had more than one content id)

A note that might not be clear, we could have done the update write queries with tt: value directly. But on average it takes five to ten seconds to find rows associated with that tt id. That means doing so would have basically brought the replication in s3 to a grinding halt. So we had to build the mapping for text table id to content id (and running the read queries took around twenty hours).

Thanks!

wikiadmin2023@10.192.32.161(dewiktionary)> select count(*) from content where content_address like 'tt%';
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (10.352 sec)

wikiadmin2023@10.192.32.161(dewiktionary)>

@Zabe or @Ladsgroup,

in German Wiktionary, someone asked why the error occurred and why, apparently, it only occurred in German Wiktionary.

Could you please explain this in a few words? Of course also in German, if you like.

With kind regards

Here's my attempt at explaining:

A script was run on all wikis a while ago to migrate the storage of revision content from one database format to another. For some unknown reason, that script failed to run properly on a few wikis, including dewiktionary.

Then, a few days ago, not noticing that failure, a second script was run to delete the old storage mechanism, assuming that the first script had succeeded. On dewiktionary that script had failed, so it deleted things that were still in use and caused this error.

@Pppery,

Thanks for the explanation. I will pass this on.

Change #1119202 merged by RLazarus:

[operations/puppet@production] logspam.pl: Consolidate the "Failed to load data blob" exception

https://gerrit.wikimedia.org/r/1119202