Adding User-notice, A deployer took a Sysadmin action to delete 2 pages to mitigate. Notice coming on wiki.

Notice given at https://da.wikipedia.org/w/index.php?diff=11489388

In T337700#8893202, @RhinosF1 wrote:

Adding User-notice, A deployer took a Sysadmin action to delete 2 pages to mitigate. Notice coming on wiki.

user-notice goes to tech news, meaning all wikis will be given notice. Too much for an issue affecting dawiki and huwiki.

Mentioned in SAL (#wikimedia-operations) [2023-05-31T18:01:21Z] <TheresNoTime> ran wikiadmin2023@10.64.32.139(huwiki)> UPDATE thread SET thread_signature = '<span title="bétaverzió"> <!--<font style="text-decoration: blink;">--><font color="red">♥</font><font color="white">♥</font><font color="green">♥</font> </font> [[User:Gubbubu|<font color="green" face="Lucida calligraphy">Γουββος Θιλο' WHERE thread_id = 1288; (with BEGIN/COMMIT) for T337700

The relevant patch is rMWf27945728f1d: In the event of preg failure in MagicWordArray throw exception but that was merged ages ago. Do we know why this started happening now? (logstash says this Monday at 9:20 UTC - doesn't match anything in SAL)

Anyhow, should we revert that patch any maybe just convert broken UTF-8 bytes to question marks?

Change 924999 had a related patch set uploaded (by Samtar; author: Samtar):

[mediawiki/core@master] MagicWordArray: Do not throw exception for `preg_match_all` error

https://gerrit.wikimedia.org/r/924999

In T337700#8893342, @Tgr wrote:

Anyhow, should we revert that patch any maybe just convert broken UTF-8 bytes to question marks?

I've partially reverted in https://gerrit.wikimedia.org/r/924999, seeing as it appears to only be preg_match_all that is throwing — I suppose � is better than an exception in these cases until we figure out what happened..?

In T337700#8893117, @TheresNoTime wrote:

Repro:

• https://da.wikipedia.org/wiki/Efficiens?action=edit&uselang=da (error)
• https://da.wikipedia.org/wiki/Efficiens?action=edit&uselang=en (working)

Affecting a significant number of pages on da.wiki, and some(?) logins

---

• mwversion: 1.41.0-wmf.10
• reqId: e0d5d384-e802-4bb3-b55c-f9ef5dba9e78
• Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   Exception: preg_match_all error 4: Malformed UTF-8 characters, possibly incorrectly encoded

exception.trace

from /srv/mediawiki/php-1.41.0-wmf.10/includes/parser/MagicWordArray.php(324)
#0 /srv/mediawiki/php-1.41.0-wmf.10/includes/parser/Parser.php(4087): MediaWiki\Parser\MagicWordArray->matchAndRemove(string)
#1 /srv/mediawiki/php-1.41.0-wmf.10/includes/parser/Parser.php(1613): Parser->handleDoubleUnderscore(string)
#2 /srv/mediawiki/php-1.41.0-wmf.10/includes/parser/Parser.php(700): Parser->internalParse(string)
#3 /srv/mediawiki/php-1.41.0-wmf.10/includes/language/MessageCache.php(1481): Parser->parse(string, MediaWiki\Title\Title, ParserOptions, boolean)
#4 /srv/mediawiki/php-1.41.0-wmf.10/includes/language/Message.php(1428): MessageCache->parse(string, MediaWiki\Title\Title, boolean, boolean, Language)
#5 /srv/mediawiki/php-1.41.0-wmf.10/includes/language/Message.php(1008): Message->parseText(string)
#6 /srv/mediawiki/php-1.41.0-wmf.10/includes/language/Message.php(1076): Message->format(string)
#7 /srv/mediawiki/php-1.41.0-wmf.10/includes/editpage/TemplatesOnThisPageFormatter.php(111): Message->parseAsBlock()
#8 /srv/mediawiki/php-1.41.0-wmf.10/includes/editpage/EditPage.php(3282): MediaWiki\EditPage\TemplatesOnThisPageFormatter->format(array, boolean)
#9 /srv/mediawiki/php-1.41.0-wmf.10/includes/editpage/EditPage.php(3211): MediaWiki\EditPage\EditPage->makeTemplatesOnThisPageList(array)
#10 /srv/mediawiki/php-1.41.0-wmf.10/includes/editpage/EditPage.php(824): MediaWiki\EditPage\EditPage->showEditForm()
#11 /srv/mediawiki/php-1.41.0-wmf.10/includes/actions/EditAction.php(66): MediaWiki\EditPage\EditPage->edit()
#12 /srv/mediawiki/php-1.41.0-wmf.10/includes/MediaWiki.php(559): EditAction->show()
#13 /srv/mediawiki/php-1.41.0-wmf.10/includes/MediaWiki.php(334): MediaWiki->performAction(Article, MediaWiki\Title\Title)
#14 /srv/mediawiki/php-1.41.0-wmf.10/includes/MediaWiki.php(925): MediaWiki->performRequest()
#15 /srv/mediawiki/php-1.41.0-wmf.10/includes/MediaWiki.php(579): MediaWiki->main()
#16 /srv/mediawiki/php-1.41.0-wmf.10/index.php(50): MediaWiki->run()
#17 /srv/mediawiki/php-1.41.0-wmf.10/index.php(46): wfIndexMain()
#18 /srv/mediawiki/w/index.php(3): require(string)
#19 {main}

That seems related to the work on T128152 and is not related to the stack trace of the inital task description. It seems also isolated to dawiki and should be fixed by T128152, but seems not worse by the code to reduce error logging.

In T337700#8893153, @RhinosF1 wrote:

Happening on even login and preferences so not LT related

I do not understand this in the context of this task, this task is LiquidThread related, it seems some corresponding work isolated to dawiki on T128152 happens, that could affect each login because it effects local mediawiki system message overwrittes.

dawiki does not use LT, huwiki does

We've been using this task for the dawiki issue as it was the same trace (although it may be a different cause as you've now pointed out - we were advised it wasn't to do with legacy encoding earlier)

Change 925003 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/extensions/LiquidThreads@master] Add truncation for database field thread_signature on insert/update

https://gerrit.wikimedia.org/r/925003

Change 925003 merged by jenkins-bot:

[mediawiki/extensions/LiquidThreads@master] Add truncation for database field thread_signature on insert/update

https://gerrit.wikimedia.org/r/925003

In T337700#8893431, @Umherirrender wrote:

In T337700#8893153, @RhinosF1 wrote:

Happening on even login and preferences so not LT related

I do not understand this in the context of this task, this task is LiquidThread related, it seems some corresponding work isolated to dawiki on T128152 happens, that could affect each login because it effects local mediawiki system message overwrittes.

Looking at it the other way around: huwiki doesn't use legacy encoding while dawiki does. So it's very very likely none of the legacy encoding work nor the LQT that started log spamming. It would be very unlikely both LQT and legacy encoding work cause breakage in two different wikis at the same time.

I'm also skeptical about legacy encoding change causing issues as I have been running this on a lot more wikis and none broke so far (including svwiki where I moved half a million pages and at least ten thousand legacy encoding pages) and dawiktionary and some more.

But if you need any help from me to debug this, I will do my best, currently dealing with another fire though.

In T337700#8893177, @TheresNoTime wrote:

https://da.wikipedia.org/wiki/MediaWiki:Templatesused — I deleted this, example page now working (https://da.wikipedia.org/w/index.php?title=Efficiens&action=edit&uselang=da)
same with:

• https://da.wikipedia.org/wiki/MediaWiki:Tog-previewonfirst
• https://da.wikipedia.org/wiki/MediaWiki:Tog-previewontop

These were all created in 2004 so whatever bug made MediaWiki mess up the encoding is probably long gone.

The LiquidThreads comment was made in 2011 and I think it was still actively developed back then so chances are that's also not a current issue.

So maybe these are rare enough that we can just fix them case by case.

I think this would find other incorrect signatures:

wikiadmin2023@10.64.48.109(huwiki)> select count(*) from thread where convert(thread_signature using binary) rlike '([\\xC0-\\xC1]|[\\xF5-\\xFF]|\\xE0[\\x80-\\x9F]|\\xF0[\\x80-\\x8F]|[\\xC2-\\xDF](?![\\x80-\\xBF])|[\\xE0-\\xEF](?![\\x80-\\xBF]{2})|[\\xF0-\\xF4](?![\\x80-\\xBF]{3})|(?<=[\\x00-\\x7F\\xF5-\\xFF])[\\x80-\\xBF]|(?<![\\xC2-\\xDF]|[\\xE0-\\xEF]|[\\xE0-\\xEF][\\x80-\\xBF]|[\\xF0-\\xF4]|[\\xF0-\\xF4][\\x80-\\xBF]|[\\xF0-\\xF4][\\x80-\\xBF]{2})[\\x80-\\xBF]|(?<=[\\xE0-\\xEF])[\\x80-\\xBF](?![\\x80-\\xBF])|(?<=[\\xF0-\\xF4])[\\x80-\\xBF](?![\\x80-\\xBF]{2})|(?<=[\\xF0-\\xF4][\\x80-\\xBF])[\\x80-\\xBF](?![\\x80-\\xBF]))';
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.239 sec)

(the regex is from here)

I've partially reverted in https://gerrit.wikimedia.org/r/924999, seeing as it appears to only be preg_match_all that is throwing — I suppose � is better than an exception in these cases until we figure out what happened..?

I think without the patch the error would still happen just much more silently but still just as broken. I dont think its an improvement to show a blank page. (To clarify, i dont think that patch will cause � to be shown - i think it would cause all text to be deleted)

If we want to silence the error, it would probably make more sense to put the text through utf8 validator's cleanup method and re-run the regex.

This is happening in the preg_match_all() branch, so it wouldn't blank the page, just ignore all magic words. But yeah cleaning it up would be nicer.

Although the other branch will, so nevermind.

Logging that I had to delete https://sv.wiktionary.org/wiki/MediaWiki:Loginsuccess as it was preventing users from logging in to sv.wiktionary

In T337700#8893830, @TheresNoTime wrote:

Logging that I had to delete https://sv.wiktionary.org/wiki/MediaWiki:Loginsuccess as it was preventing users from logging in to sv.wiktionary

sv.wiktionary also has legacy encoding and there was some work in the last days T128156: Migrate all old DB rows from windows-1252 to UTF-8 on svwiktionary, but cannot say if that is really related or just looks like that.

The LiquidThreads part of this task is fixed for the next train.

Change 924999 abandoned by Samtar:

[mediawiki/core@master] MagicWordArray: Do not throw exception for `preg_match_all` error

Reason:

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/924999/comments/3d2873db_a7a8e3da — darn!

https://gerrit.wikimedia.org/r/924999

When I checked for that page that got deleted in svwiktatiory, they were indeed moved by me. My guess is that they were invalid legacy encoding which would have not caused an error but when moved to utf-8, the check became strict which is fine in some cases but not all. I looked for any mediawiki page in nlwiki to make sure I won't break the wiki: T128154#8899826 I'll go and manually edit those 10 pages to fix their encoding.

The svwiktionary rows are:

mysql:research@s3-analytics-replica.eqiad.wmnet [svwiktionary]> select * from text where old_id in (956, 1279, 4946, 5462, 9469);
+--------+-----------------------+---------------------+
| old_id | old_text              | old_flags           |
+--------+-----------------------+---------------------+
|    956 | DB://cluster27/265098 | gzip,external       |
|   1279 | DB://cluster27/265766 | gzip,utf-8,external |
|   4946 | DB://cluster27/269404 | utf-8,gzip,external |
|   5462 | DB://cluster27/269918 | utf-8,gzip,external |
|   9469 | DB://cluster27/273899 | utf-8,gzip,external |
+--------+-----------------------+---------------------+
5 rows in set (0.002 sec)

Let me see what's in it.

I just learned logging in is still broken on dawiktionary and svwiktionary. Okay, I try to figure out what's going on. First I need to unblock the logging in in these wikis

Random note:
The latest version looks weird but it doesn't error when I try to load it:

> $blob = \MediaWiki\MediaWikiServices::getInstance()->getBlobStore()->getBlob( 'tt:9469' );

> var_dump( $blob );
string(56) "Du �r nu inloggad p� wiktionary med anv�ndarnamnet "$1"."

In T337700#8899938, @Ladsgroup wrote:

I just learned logging in is still broken on dawiktionary and svwiktionary. Okay, I try to figure out what's going on. First I need to unblock the logging in in these wikis

Random note:
The latest version looks weird but it doesn't error when I try to load it:

> $blob = \MediaWiki\MediaWikiServices::getInstance()->getBlobStore()->getBlob( 'tt:9469' );

> var_dump( $blob );
string(56) "Du �r nu inloggad p� wiktionary med anv�ndarnamnet "$1"."

Does it help to know that's https://sv.wiktionary.org/wiki/MediaWiki:Loginprompt ? (and deleting that page should unblock the logging in issue)

Yup, I ran

echo '' | mwscript edit.php --wiki=svwiktionary --user Ladsgroup --summary '[[phab:T337700]]' 'MediaWiki:Loginprompt'

and now login works

In T337700#8899829, @Ladsgroup wrote:

When I checked for that page that got deleted in svwiktatiory, they were indeed moved by me. My guess is that they were invalid legacy encoding which would have not caused an error but when moved to utf-8, the check became strict which is fine in some cases but not all. I looked for any mediawiki page in nlwiki to make sure I won't break the wiki: T128154#8899826 I'll go and manually edit those 10 pages to fix their encoding.

If the legacy encoding was windows-1252, you cant really have invalid legacy encoding, except maybe null. All byte patterns are valid. edit this was a lie, there are 5 unused bytes. I still think its pretty unlikely though as there are very few unused byte patterns

The whole thing the script did was to run iconv() on the loaded string with 'UTF-8//IGNORE' and put it back as utf-8. Either these rows were incorrectly encoded as windows-1252 or the script took a utf-8 and assumed it's window-1252. Do you want to take a look @Bawolff ?

I dont really understand. If the rows were legacy encoded as windows-1252, why would we convert UTF-8//IGNORE to UTF-8?

at the same time i see no reason why that would cause invalid unicode either.

Anyways, which script was this?

Do we know what the exact sequence of bytes were in the string causing the error?

I meant it's on windows-1252, we run iconv('windows-1252, 'UTF-8//IGNORE', $text) on it (see moveToExternal::resolveLegacyEncoding)

One of them that's causing error is here:
https://sv.wikipedia.org/w/index.php?title=Diskussion:Bardhyl_Londo&action=edit

It looks interesting:

Parsing it is fataling.

I can ask for a backup to see what exactly was stored for these entries. I'll do that on Monday.

In T337700#8900003, @Ladsgroup wrote:

I meant it's on windows-1252, we run iconv('windows-1252, 'UTF-8//IGNORE', $text) on it (see moveToExternal::resolveLegacyEncoding)

One of them that's causing error is here:
https://sv.wikipedia.org/w/index.php?title=Diskussion:Bardhyl_Londo&action=edit

It looks interesting:

Parsing it is fataling.

I can ask for a backup to see what exactly was stored for these entries. I'll do that on Monday.

Looking at the ?action=raw of that page in hexdump, its just (valid) windows-1252 encoded text (e.g. the fourth letter is 0xE5 which is å in windows-1252 and invalid in utf-8). So maybe the script didnt save or got rolled back somehow

Thanks for that. it didn't give me any error when I ran it :/ There might be something logically broken there :(

In T337700#8900028, @Ladsgroup wrote:

Thanks for that. it didn't give me any error when I ran it :/ There might be something logically broken there :(

I think there is a bug in the moveToExternal script. When processing a HistoryBlobCurStub object, it adds the "utf-8" flag without converting the character encoding. This bug appears to have been present since handling of HistoryBlobCurStub objects was added to the script many years ago in rSVN13429 (rMWc4409658c6c5b468), and it is my understanding that this is inconsistent with how MediaWiki normally loads revision text (it does convert the character encoding in this case).

HistoryBlobCurStub objects were added during the 1.5 upgrade for each revision in the cur table (the current revision of each page that existed at the time of the upgrade). When adding such rows to the text table, the upgrade1_5.php script would set old_flags to 'object' (leaving out the 'utf-8' flag) and would not convert the character encoding. (source)

In T337700#8900003, @Ladsgroup wrote:

I meant it's on windows-1252, we run iconv('windows-1252, 'UTF-8//IGNORE', $text) on it (see moveToExternal::resolveLegacyEncoding)

One of them that's causing error is here:
https://sv.wikipedia.org/w/index.php?title=Diskussion:Bardhyl_Londo&action=edit

Indeed, the latest revision (#754843) is likely to have been the current revision of that page at the time of the 1.5 upgrade.

I can ask for a backup to see what exactly was stored for these entries. I'll do that on Monday.

Especially if you did not use the --undo option of the script, a backup would probably be useful. You could determine which rows contained HistoryBlobCurStub objects, and restore them to how they were or at least fix old_flags.

Change 926658 had a related patch set uploaded (by PleaseStand; author: PleaseStand):

[mediawiki/core@master] moveToExternal: Actually convert encoding of cur_text

https://gerrit.wikimedia.org/r/926658

In T337700#8900237, @PleaseStand wrote:

HistoryBlobCurStub objects were added during the 1.5 upgrade for each revision in the cur table (the current revision of each page that existed at the time of the upgrade). When adding such rows to the text table, the upgrade1_5.php script would set old_flags to 'object' (leaving out the 'utf-8' flag) and would not convert the character encoding. (source)

Of course, $wgLegacyEncoding was added in 1.4 (rSVN5743 / rMW03e42a53d6daf8a4), so at least in theory, cur_text could be in UTF-8 even though the wiki has a legacy encoding. Is it possible to verify that this is not the case for any of the wikis? (That seems unlikely though. If the "utf-8" flag is omitted, as seems to be the case, then after the 1.5 upgrade, the revisions would have been wrongly converted, despite no conversion being needed.)

Swedish wiktionary is still broken. I can't delete pages anymore. Plus some other cosmetical errors. But I still can undelete and protect.

Logs are broken too: https://sv.wiktionary.org/wiki/Special:Logg/Taylor_49

It's probably another page in mw ns having broken encoding. The biggest hurdle is to find which one.

Okay, I fixed logs. Let me check delete.

@Taylor sorry to bother but what are the correct letter instead of the mojibakes?

Varning: Sidan du h�ller p� att radera har en historik:

Edit: nvm, found it from the official translation.

So logs and delete should be back now. Let me know if anything else is broken until we recover all old text blob addresses from backups.

This one still fails: https://sv.wiktionary.org/w/index.php?title=MediaWiki%3AHistorywarning&diff=14484&oldid=8261

Yeah, that's expected until we restore from backups.

Change 926658 merged by jenkins-bot:

[mediawiki/core@master] moveToExternal: Actually convert encoding of cur_text

https://gerrit.wikimedia.org/r/926658

OK ... T184749 ... does svwikt still contain non-ASCII-and-non-UTF8 text?

Sorta. It has content with windows-1252 encoding, a very old legacy one (before UTF-8 was adapted in the wiki) leading to all sorts of complications on edits from old times like 2004 to 2005. After the backups is restored, we will try to migrate them again to UTF-8 (the biggest reason these bugs happened was that we tried to migrate away from the legacy encoding but it had bugs in some cases.) and then we will hopefully will put an end to this edge cases and encoding madness.

YES please do decommission "Windows-1252" (and "Windows 11") ASAP.

Definitely. If all goes well, it'll be done in the next couple of weeks or so.

Looking at the backup: It indeed was an "object" and HistoryBlobCurStub object and got messed up during the change:

root@db1133.eqiad.wmnet[svwiktionary]> select * from text where old_id = 14484;
+--------+----------------------------------------------------+-----------+
| old_id | old_text                                           | old_flags |
+--------+----------------------------------------------------+-----------+
|  14484 | O:18:"historyblobcurstub":1:{s:6:"mCurId";i:5532;} | object    |
+--------+----------------------------------------------------+-----------+
1 row in set (0.001 sec)

old_id = 14484 is the text blob behind https://sv.wiktionary.org/w/index.php?title=MediaWiki:Historywarning&oldid=14484

wikiadmin2023@10.64.48.58(svwiktionary)> select * from slots where slot_revision_id = 14484;
+------------------+--------------+-----------------+-------------+
| slot_revision_id | slot_role_id | slot_content_id | slot_origin |
+------------------+--------------+-----------------+-------------+
|            14484 |            1 |           16235 |       14484 |
+------------------+--------------+-----------------+-------------+
1 row in set (0.001 sec)

wikiadmin2023@10.64.48.58(svwiktionary)> select * from content where content_id = 16235;
+------------+--------------+---------------------------------+---------------+-----------------+
| content_id | content_size | content_sha1                    | content_model | content_address |
+------------+--------------+---------------------------------+---------------+-----------------+
|      16235 |           57 | p3m94s16np2ht0qkcy7847uw6rocojc |             1 | tt:14484        |
+------------+--------------+---------------------------------+---------------+-----------------+

I'm going to revert that one back manually and see if that works

Mentioned in SAL (#wikimedia-operations) [2023-06-05T15:27:12Z] <Amir1> on s3 master: update text set old_text = 'O:18:"historyblobcurstub":1:{s:6:"mCurId";i:5532;}', old_flags = 'object' where old_id= 14484; (T337700)

It doesn't fix the issue but that can be very likely caused by the cache having old value. Now trying to invalidate the cache key

Yup, running this on svwiktionary fixed it:

$wanCache = \MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();
$cacheKey = $wanCache->makeGlobalKey( 'SqlBlobStore-blob', 'svwiktionary', 'tt:14484' );
$wanCache->delete( $cacheKey );

This is going to be fun.

I'm resetting all of the values of "object" in svwiktionary in database. That should technically fix it but we also we have a cache for a week. I'll purge for all of them tomorrow.

Done for svwiktionary, now doing dawiktionary. I don't think it's feasible to do purge the cache, in dawiki and svwiki they are over 100K of them

Change 927287 had a related patch set uploaded (by Ladsgroup; author: PleaseStand):

[mediawiki/core@wmf/1.41.0-wmf.11] moveToExternal: Actually convert encoding of cur_text

https://gerrit.wikimedia.org/r/927287

Change 927287 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.11] moveToExternal: Actually convert encoding of cur_text

https://gerrit.wikimedia.org/r/927287

Mentioned in SAL (#wikimedia-operations) [2023-06-05T22:28:36Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:927287|moveToExternal: Actually convert encoding of cur_text (T337700)]]

Mentioned in SAL (#wikimedia-operations) [2023-06-05T22:29:55Z] <ladsgroup@deploy1002> ladsgroup: Backport for [[gerrit:927287|moveToExternal: Actually convert encoding of cur_text (T337700)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-06-05T22:37:40Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:927287|moveToExternal: Actually convert encoding of cur_text (T337700)]] (duration: 09m 04s)

It's down to seven errors per hour which all are (I'm assuming) stale cache and it's hard to invalidate 100K cache keys. Should we just close this? I'm not sure. The cache expires in six days.