If I understand correctly, this regression started with the transition for the JobQueue from Redis (in which we stored serialised PHP), to EventBus/Kafka (in which we store JSON).

It seems the way EventBus encodes these parameters in JSON format, is are stricter about character encoding than MediaWiki itself.

For this particular task, it is about cirrusSearchLinksUpdate jobs (CirrusSearch\Job\LinksUpdate) that fail to execute an ad hoc "CirrusSearch\Job\ElasticaWrite" job (instantiated and run without being queued first), and upon that failure, tries to queue itself for retry. During this queueing the serialisation step in EventBus fails due to the job parameters for "CirrusSearch\Job\ElasticaWrite" containing bytes it does not accept in its serialisation format.

This could be resolved in a number of ways:

• If these invalid values come from a specific source external to CirusSearch that is known to not produce new such values, but has known-invalid values from a past version of itself (e.g. corrupt database rows for old comments only). Then that could be fixed by a one-off maintenance script that we run in prod and upon upgrades for third-parties.
• If these values aren't logically invalid within MW, and CirrusSearch does something somewhat unusual with them, then CirusSearch may need to detect, transform or skip these in some way so that they can play along with EventBus.
• If these values aren't logically invalid within MW and expected as parameters for jobs in other extensions as well, then it may be an issue for EventBus to find a way to store or tolerate these in a way that doesn't result in loss of jobs.

Checked few examples and there seem to be a mix of

• LiquidThread pages
• old pages (generally CJK languages) that do not seem to have been updated in a while (~2004)

Example of page id 220386 on jawiki:

Loading the data from from mwmaint1002 I see invalid utf8 sequences in the wikitext:

$p = \WikiPage::factory( Title::newFromID(220386 ) );
print( $p->getContentModel() . "\n" );
$po = $p->getContentHandler()->getParserOutputForIndexing( $p );
$ar = $p->getContentHandler()->getDataForSearchIndex( $p, $po, new \CirrusSearch\CirrusSearch( ) );
foreach ( $ar as $k => $v ) { if ( is_string( $v ) && !mb_check_encoding( $v, 'UTF-8' ) ) print ("BAD $k " . gettype( $v ) . "\n"); else print( "OK $k\n"); }

wikitext
OK category
OK external_link
OK outgoing_link
OK template
OK text
BAD source_text string
OK text_bytes
OK content_model
OK coordinates
OK language
OK heading
OK opening_text
OK auxiliary_text
OK defaultsort

But what I don't understand yet with this example is that we indexed this doc in the past as it is available in elasticsearch: https://ja.wikipedia.org/wiki/Wikipedia:%E5%89%8A%E9%99%A4%E8%A8%98%E9%8C%B2/%E9%81%8E%E5%8E%BB%E3%83%AD%E3%82%B0_2004%E5%B9%B411%E6%9C%88?action=cirrusDump
The the source_text is not available which means we somehow were able to send this document partially to elasticsearch... This is puzzling me as I don't see nor remember anything in the code that could have sent partial docs to elastic that could explain the presence of this data inside elasticsearch...

Tested various PHP versions hoping to find an explanation in vain.
Moving forward I think I'll add some temporary code in cirrus to detect invalid UTF-8 sequences in the source text so that we can flag and fix such pages in the elasticsearch index. Since Cirrus is refreshing its index constantly we might be able to use its data to filter the list of non-deleted pages that contain invalid UTF8 sequences in their text content.

Change 517083 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Investigate UTF-8 issues by tagging pages with problems

https://gerrit.wikimedia.org/r/517083

Change 517083 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Investigate UTF-8 issues by tagging pages with problems

https://gerrit.wikimedia.org/r/517083

Change 520185 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Fix logging of invalid UTF-8 bytes in source_text

https://gerrit.wikimedia.org/r/520185

Change 520263 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Detect and track invalid UTF8 sequences in more fields

https://gerrit.wikimedia.org/r/520263

Change 520185 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Fix logging of invalid UTF-8 bytes in source_text

https://gerrit.wikimedia.org/r/520185

Change 520263 abandoned by DCausse:
Detect and track invalid UTF8 sequences in more fields

Reason:
Does not seem to be needed

https://gerrit.wikimedia.org/r/520263

Moving to done, the number of non deleted pages that contain invalid sequence is extremely small, over all the production wikis there are:

• 3 on jawiki: https://ja.wikipedia.org/w/index.php?search=all%3Ahastemplate%3ACirrusSearchInvalidUTF8&title=%E7%89%B9%E5%88%A5%3A%E6%A4%9C%E7%B4%A2&go=%E8%A1%A8%E7%A4%BA&ns0=1
  • Wikipedia:削除記録/過去ログ 2004年11月 446キロバイト (0 語) - 2004年12月1日 (水) 00:47
  • Wikipedia:アップロードログ 2004年6月 387キロバイト (0 語) - 2004年7月2日 (金) 06:37
  • Wikipedia:アップロードログ 2004年4 104キロバイト (0 語) - 2004年4月30日 (金) 14:46
• 1 on meta: https://meta.wikimedia.org/w/index.php?search=all%3Ahastemplate%3ACirrusSearchInvalidUTF8&title=Special%3ASearch&go=Go&ns0=1&ns14=1&ns200=1&ns202=1
  • User talk:Moosh~metawiki 2 KB (0 words) - 20:24, 21 April 2015
• 1 on zhwikiquote: https://zh.wikiquote.org/w/index.php?search=all%3Ahastemplate%3ACirrusSearchInvalidUTF8&title=Special%3A%E6%90%9C%E7%B4%A2&go=%E5%89%8D%E5%BE%80&ns0=1
  • Wikiquote:删除纪录 3 KB（0个字） - 2004年12月15日 (三) 11:40