From debugging:

From a quick look at the errors in the logs there are 2 things that came up

• Failures are mostly caused in jobrunner from jobs triggered by parsoidCachePrewarm events
• The high rate of errors is caused because change prop is retrying those failures
• By querying the reqid the URI that triggers the root failure is related to bad titles

Could be related to this: https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/991664

Here is an example of a failure when creating the job:
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.01.28?id=QV8NUo0Bk70gu8GmsZtp

Also this looks like a similar issue, the titles failing have same prefix (Special:Badtitle/NS100).
For example from logs.

מיוחד:Badtitle/NS100:מצודות על מלכים ב א יג

Also from the stack trace the argument that is null causing the error is the namespace.

In T356024#9493834, @Jgiannelos wrote:

Also this looks like a similar issue, the titles failing have same prefix (Special:Badtitle/NS100).

A placeholder title like Special:Badtitle/NS100 will show up when we try to instantiate A Title with an unknown namespace ID. This would happen e.g. when the extension defining a namespace got disabled, but there are still references to that namespace in the database. This situation is bound to lead to interesting edge cases all over the place.

Wikimedia\Parsoid\Utils\Title::newFromLinkTarget should be more defensive, and should work with a "bad title". That would prevent the TypeError. It wen't fix the underlying problem though.

The root cause seems to be that a namespace was removed from hewikisource (two years ago?!): https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/752634, see T298430: hewikisource - remove "קטע" namespace.

It seems like we are trying to run jobs for reparsing pages in a namespace that no longer exists... or at least, we are reparsing pages that in some way make use of a reference to a non-existing namespace ID.

After digging a bit more on the logs it looks like there are some references from changeprop for failing requests:
https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2024.01.24?id=6PjcOY0BW1B7XSkmsiPX

The actual event that triggered the changeprop request has a title thats not correct:

{"$schema":"/resource_change/1.0.0","meta":{"uri":"https://it.wikivoyage.org/wiki/Speciale:Badtitle/NS105:Cammino_di_Oropa","request_id":"6c0e3657-8a97-46be-8507-b5f96010a6f9","id":"3850ee7d-9992-4de1-b04f-abf4c2caa7bd","dt":"2024-01-24T05:06:21Z","domain":"it.wikivoyage.org","stream":"resource_change"},"tags":["purge"]}

So the input to the jobrunner is not correct to start with. its not necessarily a problem with title resolution as shown in the stacktrace.

I think this is a better representation of logs of the lifecycle for a given failing request:
https://logstash.wikimedia.org/goto/5dce2d82ccf9d4ca0ceab11c64884efa

In T356024#9493681, @Jgiannelos wrote:

Could be related to this: https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/991664

This is not yet in production - so unrelated.

In T356024#9494204, @Jgiannelos wrote:

I think this is a better representation of logs of the lifecycle for a given failing request:
https://logstash.wikimedia.org/goto/5dce2d82ccf9d4ca0ceab11c64884efa

The url of the oldest log entries at the bottom is: /w/api.php?action=purge&format=xml... is this coming from some external script?

Yeah xml sounds suspicious.

The same doesn't apply though for hewikisource.
Here is a similar query for the hewikisource first error:
https://logstash.wikimedia.org/goto/150ade8bd1f0109d931020d48bb9887f

In T356024#9493954, @daniel wrote:

Wikimedia\Parsoid\Utils\Title::newFromLinkTarget should be more defensive, and should work with a "bad title". That would prevent the TypeError. It wen't fix the underlying problem though.

I'm not entirely sure I agree with this, as it would make us completely blind as to the underlying problem here. Most title constructors have a "throw exception" version and a "returns null" version -- the legacy parser is a little unusual in that it substitutes Special:BadTitle instead of throwing, but that's because of old legacy compatibility constraints and not something to be emulated in new code IMO.

In T356024#9494676, @cscott wrote:

I'm not entirely sure I agree with this, as it would make us completely blind as to the underlying problem here. Most title constructors have a "throw exception" version and a "returns null" version -- the legacy parser is a little unusual in that it substitutes Special:BadTitle instead of throwing, but that's because of old legacy compatibility constraints and not something to be emulated in new code IMO.

I'm happy with throwing, but it would be nice to throw a more specific exception with a more informative message.

This appears to have trailed off around 1400 on the 29th, but if there is a risk of this recurring it'd be great if we could avoid these exception spikes in future.

There is T314733 too.

I'm able to reproduce this issue on scandium with

NO_PROXY="" no_proxy="" curl --proxy scandium.eqiad.wmnet:80 "http://he.wikisource.org/w/rest.php/he.wikisource.org/v3/page/html/מצודות_על_מלכים_ב_א_יג/1211576"

https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-default-1-7.0.0-1-2024.01.30?id=IMUDXI0BySCoT0gdPgIw

Looking at the trace in the description above, the crash is in Parsoid, yes, but the issue is that a MediaWiki\Title\Title is constructed with a namespace number that Language::getNamespaces doesn't know about.

The title is created from Title::makeTitle from Title::newFromPageReference, so it gets the namespace from the page reference, which comes from a MediaWiki\Revision\RevisionStoreRecord. It's coming from the database.

I downloaded a dump of hewikisource and here's the relevant output of grep -A 10 -B 10 "מצודות על מלכים ב א יג" hewikisource-20240120-pages-meta-current.xml

  <page>
    <title>מצודות על מלכים ב א יג</title>
    <ns>0</ns>
    <id>138066</id>
    <revision>
      <id>1276519</id>
      <parentid>1211575</parentid>
      <timestamp>2021-10-02T21:26:43Z</timestamp>
      <contributor>
        <username>מושך בשבט בוט</username>
        <id>27882</id>
      </contributor>
--
        <id>27882</id>
      </contributor>
      <comment>מושך בשבט בוט העביר את הדף [[קטע:רמב&quot;ן על בראשית יד טו]] לשם [[רמב&quot;ן על בראשית יד טו]]: העברה בעקבות ביטול מרחב השם &quot;קטע&quot;</comment>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="53" xml:space="preserve">#הפניה [[רמב&quot;ן על בראשית יד טו]]</text>
      <sha1>ldefn2yf71demjfj3wju7eevp03yp65</sha1>
    </revision>
  </page>
  <page>
    <title>מצודות על מלכים ב א יג</title>
    <ns>100</ns>
    <id>425300</id>
    <redirect title="מצודות על מלכים ב א יג" />
    <revision>
      <id>1211576</id>
      <timestamp>2021-09-26T17:49:49Z</timestamp>
      <contributor>
        <username>ShalomOrobot</username>
        <id>28707</id>
      </contributor>
      <comment>ShalomOrobot העביר את הדף [[קטע:מצודות על מלכים ב א יג]] לשם [[מצודות על מלכים ב א יג]]: העברה בעקבות ביטול מרחב השם &quot;קטע&quot;</comment>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="55" xml:space="preserve">#הפניה [[מצודות על מלכים ב א יג]]</text>
      <sha1>q2a2beryjv8vohgo2nxriz7hdg3cxpn</sha1>
    </revision>
  </page>

The revision id from the second page is what's used above.

<ns>100</ns>

That's the problem. That namespace doesn't exist, and hasn't for two years. T298430#7608752 sais that all pages have been moved away from the namespace, but apparently, that's not true.

The fix is to manually change the namespace ID of the page to a different namespace.

We should catch this situation waaayyy earlier, and avoid generating jobs for bad pages.

What does grep '<ns>100</ns>' | wc -l look like? Curious to know how big of a cleanup this is.

Change 994360 had a related patch set uploaded (by Arlolra; author: Arlolra):

[mediawiki/services/parsoid@master] Assert invariant when constructing a Title from a LinkTarget

https://gerrit.wikimedia.org/r/994360

In T356024#9500625, @Arlolra wrote:

> grep "<ns>100</ns>" hewikisource-20240120-pages-meta-current.xml | wc -l
   42922

The database agrees:

MariaDB [hewikisource_p]> select count(*) from page where page_namespace = 100;
+----------+
| count(*) |
+----------+
|    42922 |
+----------+

They are all redirects, though:

MariaDB [hewikisource_p]> select count(*) from page where page_namespace = 100 and page_is_redirect=0;
+----------+
| count(*) |
+----------+
|        0 |
+----------+

Change 994360 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Assert invariant when constructing a Title from a LinkTarget

https://gerrit.wikimedia.org/r/994360

I'm a sysop in the hewikisource. If this makes it easier, you can restore this namespace, and I will run a bot to delete all these pages. I prefer if you can delete them from the database itself instead, but it can be an option.
To prevent the next possible issue, I suggest to delete automatically every page while namespace is deleted (actually we thought that this is the status already, that's why we haven't deleted these pages before), or at least to ensure that all the pages in the namespace have been deleted.

Change 997591 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.19.0-a16

https://gerrit.wikimedia.org/r/997591

Change 997591 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.19.0-a16

https://gerrit.wikimedia.org/r/997591