I figured out this is due to ClassicInterwikiLookup::getAllPrefixes() never returning interwiki prefixes from virtual domains, which, in turn, makes the siteinfo API module never return these prefixes, and so Parsoid never fetches them from the API. Which is odd, because ClassicInterwikiLookup::fetch() does fetch interwiki prefixes from virtual domains, and SpecialInterwiki has its own fetching of interwiki prefixes from virtual domains.

I have now fixed this for ourselves using a new implementation of InterwikiLookup that adds prefixes from virtual domains. Should I open a new task for this?

Adding the Interwiki and API projects because the root causes of this bug are:

• ClassicInterwikiLookup::getAllPrefixesDB() (which does not account for virtual domains at all)
• ApiQuerySiteInfo (which does not return all interwiki prefixes, because it uses InterwikiLookup)
• SpecialInterwiki (where the lack of interwiki links from virtual domains is treated as a feature, because local and global interwiki prefixes need to be displayed separately)

It seems that InterwikiLookup::getAllPrefixes() will need an interface change to accomodate both ApiQuerySiteinfo's use case (return all prefixes including those from virtual domains), as well as SpecialInterwiki's (return only interwiki links from a specific virtual domain, or only locally defined ones).

The terminology here is also a bit problematic, because getAllPrefixes accepts one parameter called $local, which refers to the iw_local field in the interwiki table, and not specifically fetching local interwiki links (as opposed to global interwiki links, from the virtual domain). I propose a flag parameter called $from, and flags InterwikiLookup::FROM_GLOBAL_INTERWIKI, InterwikiLookup::FROM_GLOBAL_INTERLANGUAGE and InterwikiLookup::FROM_LOCAL. It will have a default value including all these domains, and SpecialInterwiki can then call the method three times to fetch all three domains.

I don't think ClassicInterwikiLookup::getAllPrefixesPregenerated() will be affected at all, because presumably virtual domains aren't used in that case at all.

Thanks for your investigation. It doesn't look like a Parsoid issue, but more in the other components. I recommend retitling and repurposing the phab task for the actual issue in those components. I am going to untag our team here for now.

I updated the task description to explain how it's a siteinfo API issue, and moved it back for triage on the API Platform Team board since it's no longer "other teams work".

Thanks for moving it back, @KockaAdmiralac ! We will do some additional investigation to see if we can figure out what's happening with more clarity. For some context, it looks like the Interwiki extension is currently unmaintained and might be slightly out of our wheelhouse if it looks like something in the extension is the root cause.

Also -- do you know if this might be the result of a recent change? Have you used this capability previously, and it just recently stopped working? Mostly just curious since knowing when it started might help us work back to the root cause.

I bolded the answers to your concerns below.

The Interwiki extension was merged into core in MediaWiki 1.44, about 1.5 years ago. As a result of that, all related code is now in core, so I don't think you will have to touch unmaintained extensions.

As a result of the merge, the variables $wgInterwikiCentralDB and $wgInterwikiCentralInterlanguageDB have been removed in favor of virtual domains virtual-interwiki and virtual-interwiki-interlanguage. These variables/virtual domains are used for specifying the global database for interwiki/interlanguage links, such that multiple wikis can share interwiki and interlanguage links. (I, in particular, use these virtual domains between Undertale and Deltarune wikis to share common interwiki links.) This task is about how ApiQuerySiteinfo module, which essentially uses InterwikiLookup::getAllPrefixes() under the hood, does not respect these virtual domains and returns a list of only local interwiki.

As far as I can tell, ClassicInterwikiLookup::getAllPrefixes() never respected either the variables or the virtual domains, so this isn't a result of a recent change. Parsoid, however, relies on that method to build a bi-directional interwiki map (see T270444: Parsoid needs a bidirectional interwiki map (and hooks)) used when parsing wikitext. From the perspective of ApiQuerySiteinfo, adding this would be a feature. From the perspective of Parsoid, adding this would fix a bug. On Wikimedia wikis this bug never manifested because you use pregenerated interwiki prefixes. On other wikis this bug never manifested because nobody else uses Parsoid for read views combined with interwiki virtual domains (or Parsoid for read views in general, I guess).

Before the Interwiki extension was merged into core, implementing support for global interwiki databases in getAllPrefixes() was pretty much impossible without adding a dependency on that extension. However, now that the virtual domains have been introduced (and the InterwikiLoadPrefix hook has been hard deprecated), it makes sense for getAllPrefixes() to query the virtual domains as well.

The reason why I think an interface change to getAllPrefixes() is needed is that I already implemented a workaround which replaces ClassicInterwikiLookup on our two wikis with our own implementation, in which getAllPrefixes() queries the virtual domains as well, and merges the results. This fixes the Parsoid bug, but does not play well with SpecialInterwiki, which relies on the fact that getAllPrefixes() returns exclusively local prefixes. To display the global interwiki tables, SpecialInterwiki does its own queries to these virtual domains. This manifests as a different bug, where Special:Interwiki now displays duplicate interwiki prefixes in the local table, but doesn't allow admins to edit the prefix if they hit the button on what is actually a global prefix. I think SpecialInterwiki should not concern itself with interwiki lookup, and that these queries should be the responsibility of InterwikiLookup instead.

I can implement a patch for this, if seeing the proposed change makes it clearer what the task is about.

Thank you for the explanation! I really appreciate it. I'm still relatively new, so I wasn't aware of the transition from the extension into core either -- apologies for the confusion there.

Submitting a patch for this would also be absolutely welcome. While I'm not sure we have the capacity to dig into fixing this immediately on the team, we can certainly prioritize code review for your fix if you'd like to take a crack at it. :)

Change #1315813 had a related patch set uploaded (by KockaAdmiralac; author: KockaAdmiralac):

[mediawiki/core@master] Interwiki: Let getAllPrefixes return prefixes from virtual domains

https://gerrit.wikimedia.org/r/1315813