In https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1308173 it was proposed to only wrap the title (not the namespace) in language tags. I don't think this is correct as-is, but it may be worth recognizing "canonical" namespaces specifically and adding language=en dir=ltr to those, to properly deal with that specific corner case.

Filed T431358: Use PROTO_CANONICAL instead of PROTO_RELATIVE in ExpandRelativeAttrs and T431359: Parsoid should consistently use `_` instead of ` ` in wikilink URLs as action items.

My recollection is that it wasn't just TOCs, a lot of skin code and user gadgets also assumed that the base href of the page was unmodified. Setting the base href to something "unexpected" broke a *lot* of things on subpages; the TOC links were just the thing which got written into the phab task.

Yeah, like I said, it was added for DiscussionTools and I could easily agree that it was added in the wrong spot in retrospect. But IIRC it was added there at the time specificially because it *didn't* affect as many places as WikiPage and so seemed 'safer'.

Verified that https://test.wikipedia.org/wiki/User:Cscott/T387374 seems to work with -a14

Above patch should fix the remaining failures. If I can get it reviewed, I'll backport it in the morning window.

In retrospect it is likely the ArticleParserOptions hook should have been added to WikiPage instead of Article, but as I understand it the migration to WikiPage was never completed. Perhaps by the time that Article is finally deprecated and removed we can deprecate and remove the ArticleParserOptions hook as well, but in the short term while Article (and ArticleParserOptions) exists, it seems like the right move is to use it.

In T429408#12077110, @ssastry wrote:

The ideal solution for previews (and other places where we do wt2html transforms) is to ensure that any code constructing a ParserOptions object consults ParserMigration to set the useParsoid flag in the parser options object. Currently, this happens via the onArticleParserOptions hook handler in the ParserMigration extension. *However*, in a number of places in core and extensions, a parser options object is created via the WikiPage::makeParserOptions call where there is no Article object present, and the onArticleParserOptions handler signature requires an Article (which is used by some extensions like DiscussionTools).

WRT the root issue here, https://www.wikidata.org/w/rest.php/v1/revision/2512536582/html is asking for https://www.wikidata.org/w/index.php?title=Q20826935&action=info which is content model "wikibase item" and I don't think the REST API was every supposed to support returning HTML content for titles which were not "wikitext" content model. I'm consulting with folks on slack to figure out what that query is "supposed" to return, but it seems to be undocumented behavior.

In T430778#12075841, @Jgiannelos wrote:

The initial patch wont fix the production issue. Here is some more details after debugging. From the parseroutput API we can get details about what the parseroutput has for the failing page:
https://www.wikidata.org/w/api.php?action=parse&format=json&formatversion=2&page=Q21039773&parser=parsoid&prop=parseroutput

More specifically, we can see that the check if the title exists fails because:

• The parseroutput title is bogus: "Badtitle/dummy_title_for_API_calls_set_in_api.php"
• The namespace is -1 (special)

But the revision id is valid so eventually after the fix the getRevisionById would also fail (as @neriah pointed out)
I sent a follow up patch that puts a guard on the lookup as well.

It's possible that we could support the horrible enwiktionary hacks, now that we are also putting heading information into strip markers...

Ok, the title is a translatble part of the page, when the translate extension is installed, and the translation is done by a pseudo-section in the translations called "Page display title". You can edit the translation like: https://www.mediawiki.org/w/index.php?title=Special%3ATranslate&group=page-Content+Transform+Team&action=page&filter=&action_source=translate_page&language=es
and the bug is coming up when the title has *not* been explicitly translated, and so translate is attempting to fall back to the "real" page title -- instead of just not setting displaytitle in the first place, which is probably what it *should* do when no translation exists.

I think the translate extension is getting confused, since https://www.mediawiki.org/wiki/Content_Transform_Team/de contains the correct (translated) name while https://www.mediawiki.org/wiki/Content_Transform_Team/es does not.

It would be nice to allow @tstarling to participate remotely in this session from Sydney, Australia, and having this session between 11am-3pm Paris time would be most likely to let him do that.

Part of the problem here is that tokens are not really meant for direct conversion back to source text. We could use DSR ranges to extract the source text (assuming accurate Source values are maintained), and the PFragment-extension-tags patch series would also solve this by not tokenizing transclusion arguments until they are actually used, so we'd have access to the raw text still. But of course the raw text a nested extension tag gets is not guaranteed to be related to anything directly typed by the editor, nor to the value the extension gets: {{#tag:pre|{{{1}}}}} might be the extsrc but it's not the value that would actually be parsed after the source was expanded.

1. Risky Patch! 🚂🔥

Another case, falling out from T429624, {{#tag:references|<ref name="a">a</ref><ref details="p1" name="b">b</ref>}} generates "extsrc":"{{#parsoid\u0000fragment}}{{#parsoid\u0000fragment}}" because the internal ExtTagPFragment tokens leak into the extsrc attribute.

Elevating priority because this breaks 3rd party login (including wmf cloud login) to Wikimedia servers for all downstream apps using the mediawiki/oauthclient package (and there's quite a lot of dependencies of that in https://packagist.org/packages/mediawiki/oauthclient/dependents?order_by=downloads ).

global search uses wikimedia/toolforge-bundle which uses mediawiki/oauthclient. And mediawiki/oauthclient uses format=json wouldn't you know.

On IRC it was said:

if its a oauth1 user, it appears there was a bit of a surprise breaking change
bawolfs comment in -tech

One option would be to *move* the fragment from the meta tag to the indicator, instead of *copying* it. Or at least removing the id from the meta tag once it is copied? (There's probably some issue here about DOM->HTML->DOM conversions which is why this is being cloned even without an explicit fragment clone operation.)

This keeps surfacing every couple of years, mostly related to wanting to use parsoid IDs as stable element IDs (T116350: Design and implement an algorithm to provide stable element ids). Our current solution to this use case is T149667: Build an article annotation service which does the mapping between elements in different revisions (or different parses of the same revision) without requiring full determinism from the wikitext stack.

At the risk of throwing fuel on the fire, I'll note that extension tag syntax is deliberately designed[1] for embedding non-wikitext content inside wikitext, and so syntactically it deliberately ignores anything except the closing </ref>. There's some discussion about this here https://en.wikipedia.org/wiki/User:Cscott/Ideas/A_Dozen_Visions_for_Wikitext/Extension_tag_fragments as a proposal to both allow nesting and make it /actually/ able to contain arbitrary non-wikitext content (T393795, which has some initial support already).

I could easily add a check for extension.json and skin.json. That's more MediaWiki-specific, but this is a MediaWiki-specific plugin at this point (living in mediawiki-phan-config) anyway.

I forgot to backport this when I backported the two other patches (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1305018 and https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1305025 ) which depend on this.

[[parser] Rename mStripExtTags to useParsoidFragments (1305223) · Gerrit Code Review](https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1305223) will fix this.

I'll fix this, it's my fault. :(

Note that the existing legacy behavior is to render this as -{Джордан}- -- that is, the actual -{ and }- brackets are rendered as well as the contents. The requested Parsoid behavior doesn't exactly match legacy behavior.

(Currently blocking all gate-and-submit to core)

Now failing for mediawiki-core as well, eg https://integration.wikimedia.org/ci/job/mediawiki-node24/39102/console

I don't agree. I believe we are getting corruption due to the network problems. The corruption eventually causes npm to error out. Clearing the caches is a brute-force workaround to the problem of npm downloaded corrupted (truncated?) files from the buggy network connection.

@thcipriani how do you remove the cache for a job?

I think I just saw this in a patch to mediawiki-core as well?

18:03:21 npm warn tarball tarball data for @typescript-eslint/eslint-plugin@8.46.0 (sha512-hA8gxBq4ukonVXPy0OKhiaUh/68D0E88GSmtC1iAEnGaieuDi38LhS7jdCHRLi6ErJBNDGCzvh5EnzdPwUc0DA==) seems to be corrupted. Trying again.
18:03:31 npm error code ENOENT
18:03:31 npm error syscall stat
18:03:31 npm error path /cache/_cacache/content-v2/sha512/e4/7b/c83c8c01523a98983d7c98f4d2f65b38138df7250014c4374962a0a925d0789d457c0b931919e7c18a5503cb3d85f80ff2d0a6af8650a858788c38f1e02f
18:03:31 npm error errno ENOENT
18:03:31 npm error enoent Invalid response body while trying to fetch https://registry.npmjs.org/zip-stream: ENOENT: no such file or directory, stat '/cache/_cacache/content-v2/sha512/e4/7b/c83c8c01523a98983d7c98f4d2f65b38138df7250014c4374962a0a925d0789d457c0b931919e7c18a5503cb3d85f80ff2d0a6af8650a858788c38f1e02f'
18:03:31 npm error enoent This is related to npm not being able to find a file.

in https://integration.wikimedia.org/ci/job/mediawiki-node24/39071/ on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1271913

Extension:Chart returns a raw HTML string from the parser function with (from ParserFunction.php):

		return [ $html, 'noparse' => true, 'isRawHTML' => true ];

support for which was added in Icb8eae for T381617. This gets tunneled through as a "nowiki" strip marker (but carefully disambiguated from an "actual" <nowiki>) and returned to Parsoid as a HtmlPFragment... which then gets embedded as a {{#parsoid\0fragment:...}} and tokenized as a a DOMFragment token via TokenizerUtils::parsoidFragmentMarkerToTokens(), wrapped in a new SelfclosingTagTk( 'template', [], $dp );.

findVariantLink seems like exactly the right solution to me, and is how category links are resolved in the LinksUpdate jobs. @Winston_Sung can you explain why you think that wouldn't work?

In T209394#12038075, @Tacsipacsi wrote:

In T209394#12037436, @cscott wrote:

When run in mediawiki-core, a set of MediaWiki\* namespaces would be friends, but a call to an @internal method of Wikimedia\Parsoid would be flagged.

Not all of it – MediaWiki\Extension\* and MediaWiki\Skins\* should definitely not be friends, but there are also some extensions and skins that don’t have the Extension/Skin subnamespace, e.g. Babel chose MediaWiki\Babel\* and Minerva Neue is MediaWiki\Minerva\*.

There might be some related commentary in https://pt.wikipedia.org/wiki/Wikip%C3%A9dia:Esplanada/geral/Se%C3%A7%C3%A3o_Refer%C3%AAncias_(19jun2026)?

https://pt.wikipedia.org/wiki/Encounter_at_Farpoint and https://pt.wikipedia.org/wiki/601_Lexington_Avenue appear to work now?

Several pages have additionally been reported with broken <ref> group resolution in notes sections, possibly as a secondary consequence of Parsoid restructuring section scope.

If the section is expected to be in the table of contents, it should be in wikitext format, == ... ==, and that ought to imply that is is an editable section and also collapsable. Why is it so important that the section edit link be hidden for this section?

Seems related to T424391: Parsoid Language Converter doesn't convert rules inside -{ ... }- markup. Should be fixed in the core language converter implementation (T380517) not the Parsoid implementation, which will be removed shortly (T424409).

Should be re-examined in light of T424408: Use new ParsoidLanguageConverter pass for REST page API and the recent "Shadow pages" support.

Done as part of T424773: __NOCONTENTCONVERT__ is not honored in Parsoid and T373384: Parsoid doesn't properly handle double-underscore magic words.

In T424370#11856780, @Djido wrote:

BTW, NOCONTENTCONVERT does not work in Parsoid

I'm surprised that the proposed standard doesn't use the https://en.wikipedia.org/wiki/Well-known_URI RFC.

Implemented a version of this check; described in more detail at https://www.mediawiki.org/wiki/Continuous_integration/Phan/Plugins

Per-namespace @internal would work if we could define a set of "friend" namespaces. That is, when checking Parsoid, all namespaces under Wikimedia\Parsoid\* are considered "friends" and can call @internal methods defined in Wikimedia\Parsoid\*. (That is, Wikimedia\Parsoid\Wt2Html is a friend of Wikimedia\Parsoid\Utils and they can call each others @internal methods.) However, a call from Parsoid to an @internal method of \MediaWiki\Title (something outside the friend universe) would be flagged.

The patches were tagged with T417531: Section wrapping should use precise information about HTML-syntax headings

Ok, the TemplateData extension looks for the section edit link and does a regex substitution on this, so this might actually be a subcase of the T422291: Wrong section edit link target with Parsoid when section is transcluded bug: if the section edit link is corrected, then the magic being done by the TemplateData extension might actually work.

The link to edit the templatedata is part of the content added by the <templatedata> extension tag, so it appears something is going wrong with the context we are giving that extension.

In T422291#12024000, @Repakr wrote:

It also has wrong links in Template data's edit button. The first image is Parsoid, the second image is the legacy parser. The example is from ukwiki.

Could you provide a link to a page which demonstrates the "Then there is a problem with TemplateData that also has an invalid link if you try to edit it from the documentation box." issue?

In T429038#12032359, @Repakr wrote:

Could we revert the deployment of Parsoid for ukwiki for now, before this ticket T422291 is fixed? Or prioritize resolving the ticket? Because a lot of documentation pages are broken, we prefer to have an edit link in the section, unlike enwiki, which has them only at the top of the documentation box. Then there is a problem with TemplateData that also has an invalid link if you try to edit it from the documentation box. And now users are complaining on the page Requests for administrator attention (https://uk.wikipedia.org/w/index.php?title=%D0%92%D1%96%D0%BA%D1%96%D0%BF%D0%B5%D0%B4%D1%96%D1%8F%3A%D0%97%D0%B0%D0%BF%D0%B8%D1%82%D0%B8_%D0%B4%D0%BE_%D0%B0%D0%B4%D0%BC%D1%96%D0%BD%D1%96%D1%81%D1%82%D1%80%D0%B0%D1%82%D0%BE%D1%80%D1%96%D0%B2&diff=48222382&oldid=48220768 Section «Захищена сторінка») that they cannot edit discusions on the page https://uk.wikipedia.org/wiki/%D0%92%D1%96%D0%BA%D1%96%D0%BF%D0%B5%D0%B4%D1%96%D1%8F:%D0%9F%D0%B5%D1%80%D0%B5%D0%B9%D0%BC%D0%B5%D0%BD%D1%83%D0%B2%D0%B0%D0%BD%D0%BD%D1%8F_%D1%81%D1%82%D0%B0%D1%82%D0%B5%D0%B9 through edit links, becuase the transcluded through template https://uk.wikipedia.org/wiki/%D0%A8%D0%B0%D0%B1%D0%BB%D0%BE%D0%BD:Rename.

The relevant wikitext is:

{{notelist|group=lower-roman|refs=
{{efn-lr|name="NB_Herkon"|Historically, the term ''Herkon relay'' was occasionally referred to as synonymous to ''reed relay'' even in the literature. "Herkon" was actually a [[Standard Elektrik Lorenz|SEL]]/[[International Telephone and Telegraph Corporation|ITT]] trademark standing for "[[hermetic seal|hermetically sealed]] contact".<ref name="Scheidig_1959"/><ref name="Steinbuch_1962"/><ref name="Schönemeyer_1964"/><ref name="Oden_1964"/><ref name="Steinbuch-Wagner_1967"/>}}
}}

which ultimately desugars into something like:

{{#tag:references|<ref name=...>...{{#tag:ref:...|name=foo}}</ref>}}

and then the later references section includes those embedded references from the notelist.

This is a ref-in-ref issue: the backlinks should point to an entry in the "Notes" section, which is itself a <ref>.

I guess, if the -{ and }- were <b> and </b> we wouldn't have any problem saying that this was bogus, similar if this were [[ and ]]. I'd suggest we fix the wikitext.

I'm not sure this isn't acceptable behavior for Parsoid. There's got to be some limit on what sort of construct you can get away with encapsulating with -{ ... }-. I think it's reasonable to say that the contents should be "balanced".

Minimized test case:

{{#tag:mapframe||text={{#tag:pre|San Francisco}}|width=350|height=350|zoom=13|longitude=-122.39953994750977|latitude=37.81032643553478}}

The {{#tag:pre}} returns a strip marker, which is then passed as the value of the text attribute to the <mapframe> and Parsoid loses track of it.

See also T122390: Is RDFa metadata in Parsoid HTML head actually useful to you / no user name & edit comment suppression in Parsoid <head> metadata for a historical discussion of the use of the HTML <head>; obviously time has passed since then and folks have new needs.

@Od1n DiscussionTools uses a fairly-complicated heuristic to determine whether a page "could be a discussion" since some discussions happen outside of the Talk: namespace. If you give specific examples I could probably check to see exactly what is triggering the heuristic in that case.

If we create {{refbegin2}} without the -{ and define (for transition purposes) {{refbegin}} as {{refbegin2}}-{ then editors can just use {{#subst:refbegin}} when they initially author the template. That also avoids "breaking" pages while the bot is running. Does that sound reasonable?

The primary disadvantage is that this requires exposing to the parser (parsoid) not only the "default" image size but also the small and large sizes in order to add all of the correct classes. I'd prefer not to add that complexity if we can avoid it; this reduces entanglement between the core parser and the thumbnail/media system.

This was (finally?) done in ef8dfcf26bdbc74a0e59ccd275b8dd88c039fee5.

It turns out that some folks do still require noreferrer? T429090: Add "noreferrer" to the "rel" attribute for links leading to archive.today or one of its mirrors

This was done for T427561: Remove outdated rel=noreferrer from target=_blank links created by $wgExternalLinkTarget and WikimediaCustomizations would add rel=noreferrer back to legacy parses but not to Parsoid Read Views (almost all wikipedias).

Current status: SerializationTestTrait: introduce ::normalizeForObjectEquality (1300886) · Gerrit Code Review exists but has conflicts that need to be resolved; that's the prerequisite for both of the above patches.