I need to dig into PHP semantics. Is it possible you might be suppressing the notice on $parent->children because of the =& instead of plain =? (line 246)

Did you check the other issues from https://phabricator.wikimedia.org/T206940#4670526 ? Sounds like you're saying that wt2html is fixed (both PHP and Parsoid agree and do something sensible) but that html2wt for video specifically is still broken since it treats the (invisible) embedded alt attribute as plaintext rather than HTML. Or maybe html2wt is alright but we shouldn't be embedding the invisible alt as HTML but should be doing the same tag-stripping that we would do for a "real" alt attribute. (I think I'm a little partial to this latter.)

Copied from a discussion at https://gerrit.wikimedia.org/r/#/c/mediawiki/services/parsoid/+/467531/6/lib/wt2html/tt/LinkHandler.js@1014 wrt how [[File:Foo.jpg|{{sometemplate}}]] gets parsed:

In T208549#4716336, @Joe wrote:

• I didn't check the size of the bytecode cache databases to see if that exploded. If that's the case, I might want to purge it today.

A bunch of new messages are getting tidied and going through remex now ( https://gerrit.wikimedia.org/r/#/q/topic:deprecate-wgtidy ) -- although that *shouldn't* cause problems, here are a couple of wild theories:

1. MessageCache is missing somehow ( 4b1db1190bb8f2a115c6a81a5ee487b7d18cd303 ) and tying up CPU (but I'm pretty sure the misses would show in logs). Rolling back would cause misses on the updated messages so "not solve the issue".
2. ParserCache is missing somehow ( 58abac2d1489cdfaaf2ffdf2f9e1214509760b31 ) but I'm pretty sure that would show up in analytics. Same thing, roll back could cause misses of the previous misses.
3. Some system message is triggering some weird infinite-loop bug in Remex. Of course, remex has been used for tidy in all article content for months now, but I'm always ready to be surprised. If there's a specific URL which triggers the CPU hog behavior, that would be a big clue. But wouldn't explain why rolling back didn't help.
4. It's not my fault at all. (I like this one the best, but it doesn't help you fine folks in ops any.) ;)

SEO optimization came up on the Audiences 1 QCI presentation, and it was mentioned that one question we had was whether Google used the same ingestion pipeline for all languages / wikis, or whether there were certain things that would work differently on (say) English wikipedia -vs- Spanish wikisource.

I think I could be satisfied with a good documentation comment for this method elaborating these points:
(a) state explicitly that this uses the content language, not the user interface language
(b) is intended for parser functions and tag hooks which appear in article content, not UX (special pages, warnings, etc)
(c) the result will be subject to language conversion (which is unusual for system messages, which are more usually pre-converted to a specific variant)

See also T114640: RFC: make Parser::getTargetLanguage aware of multilingual wikis, which goes in to more detail about the problems one has in general when trying to make "user interface elements" from "content" markup.

I'm not convinced the helper should be in Parser.php. Parser functions and tag hooks are a bit of a special case, since they actually are in the content language. In 95% of the cases where system messages are used, they should be in the user interface language -- compare https://codesearch.wmflabs.org/search/?q=addWikiTextAsContent with https://codesearch.wmflabs.org/search/?q=addWikiTextAsInterface for example. Even when you do want the content language, it's probably because the thing you're adding would be better off as an interface message, for example see bacd87e4942baa34808a1b77d3b29bfdb566cc17.

Verified that rETRAb2586aebd94d: Avoid untidy calls to OutputPage::addWikiText() didn't make it to the REL1_32 branch of Translate, and so no further backporting should be needed.

I think you still need to sanitize -- for example, the <script> tag has its payload delivered by the node children, not the attributes. Potentially <style> is dangerous in the same way, if there is a CSS vulnerability. But again, a tag whitelist should be sufficient for this.

Patch at https://github.com/wikimedia/language-data/pull/37

w00t! Now we can do https://gerrit.wikimedia.org/r/368248 (T117845) ?

There's a lot of backlog to read through but -- yeah, Wikimedia has *always* used their own patched version of tidy. The stock debian libtidy has never exactly matched the WMF version, although it's possible the precise differences weren't previously well-covered by parserTests. All use of libtidy is deprecated and is being removed ( https://gerrit.wikimedia.org/r/467972 ) so it's just a matter of keeping our tests of long-term-supported releases sane by maintaining access to the WMF-patched-version-of-old-tidy.

Thanks, @Legoktm!

OK, two patches to fix the issue: belt and suspenders. Both of them are potential candidates for cherry-picking to 1.32, but let's get them merged on 1.33 first.

I bet I technically have all the required permission bits to do this myself -- looks like @Legoktm made the last release -- but it would be helpful to have some instructions, either on-wiki or in the README, since I've never published a composer package before.

Ah, ok, thanks! That means I don't have to worry about this being an "unbreak now" sort of bug.

Arrrrgh. WikiBase/lib/includes/LanguageWithConversion.php contains code cut-and-pasted from mediawiki-core. Anyone want to guess the odds that it wasn't updated when the code from core was updated?

Ok, merged T207447: uselang=zh-hant-hk causes fatal exception of type "BadMethodCallException" into this one, because the stack trace is pretty much identical:

Also -- sr-cyrl, zh-hans-tw, etc are not actually the mediawiki-internal names for these languages. https://gerrit.wikimedia.org/r/460039 would have added support for using the standard names, but I'm a little bit surprised that anything is generating links to these "non-mediawiki" (but BCP 47 standard) codes. I mean, it's good -- we *should* be trying to move to using the proper BCP 47 codes -- but it's worth trying to track down where these links are coming from, since they wouldn't have worked prior to 460039.

Given that uselang processing is involved, perhaps https://gerrit.wikimedia.org/r/460039 is the proximate cause. Can we get a stack trace for that exception?

Re question #3 wrt JS-vs-PHP: it's important to understand the role of the social ecosystem involved. One reason why markdown is widespread (and wikitext is not used at all outside the WMF environment) is that we've never had a really good/fast standalone parser. Even our own WMF research department uses mwparserfromhell (written in Python) instead of either of the two "official" WMF parsers. (And they probably won't migrate to an official parser even once/if Parsoid is in PHP.)

Re the first question, I feel the "significance" section presupposes a particular answer (ie, that WYSIWYG editors aren't appropriate for templates, and so wikitext editing should be limited). I don't agree (T114454), but regardless, here's my attempt at rephrasing the prompt in a neutral manner:

Any updates on this task? As I said, I'd like to write Scribunto/JS using v8js at some point. If v8js would be helpful to wikibase, perhaps we can pool efforts.

It's not happening in 1.32, we just branched that. Hopefully for 1.33!

I filed T207297: Phan SecurityCheck-XSS and SecurityCheck-SQLInjection errors in SecurePoll extension out of an abundance of caution, but it's probably a dup. Slightly different errors, though...

@Osnard Could you see if my updated patchset works for you? Rather than explicitly test reverseMwApiMap again, I just unconditionally called ParsoidConfig#getPrefixFor() to set the prefix (even if it was already set). That's a little more consistent with the direction we really want to go (T206764: Remove `prefix` from Parsoid and use `domain` consistently as configuration key).

In https://gerrit.wikimedia.org/r/446766 I introduced BabelLanguageCodes::getCategoryCode() which maps mediawiki-internal language codes to appropriate category names. The current algorithm is to use the (lowercased) mediawiki internal code if it doesn't contain a hyphen (eg en, simple, de), otherwise use the properly-capitalized BCP 47 code (zh-Hans, etc). This matched previous expectations as canonized in the extensions phpunit tests. If we wanted some other behavior for category codes it ought to be straightforward to patch getCategoryCode() for whatever is desired.

The patch is merged, but we're going to need a new version of remex released to composer and mediawiki-core updated to require the new version before this bug is actually fixed in production uses of php 7 (ie, when running the php 7 jenkins tests).

We only emit the "@graph" form in purtle if the API is used to annotate more than a single entity: https://github.com/wikimedia/purtle/blob/master/src/JsonLdRdfWriter.php#L37

In T189966#4323531, @tstarling wrote:

• Special CDN integration with the Key header (formerly X-Vary-Options). X-Vary-Options was a feature I introduced to avoid splitting the CDN cache between IE and everything else, which were sending slightly different Accept-Encoding request headers (differing by a space)

This turned into a little rathole, but I've come out the other end fixing (a) how Parsoid parses alt/link options (wikitext markup including <nowiki> is allowed), (b) how Parsoid renders alt/link options (consistent stripping), (c) how core renders link options (<nowiki> expansion and stripping consistent with alt) , (d) how core handles ampersands in alt/link options (bug in remex), and (e) how Parsoid handles ampersands in alt/link options. Now we've just got to get those three patches merged, starting with the remex bug (T207088: Remex double-decodes HTML entities on PHP (not HHVM)) because the newly-added test cases won't pass on jenkins until remex is fixed and the fix is packaged and released so composer can get to it.

Turns out there's a bug in how core PHP parses [[File:Foo.jpg|link=Foo''s bar''s]] (which is a valid title) or [[File:Foo.jpg|link=''Main Page'']] (where the italics apparently should be stripped). So now I've got a patch for core as well as one for Parsoid...

Well, a combination of things. It's triggered by the quote marks, but then we're not handling the <nowiki> in the serialized version either. [[File:Foo.jpg|alt=<nowiki>''alt''</nowiki>]] ought to be the "correct" way to get embedded single quotes into the alt value. It looks like we might have a similar issue with the link option as well. I've got a working patch for alt, working on understanding the link issue.

i'm working on a patch

Hm, line breaks appear to be a red herring, see: https://en.wikipedia.org/w/index.php?title=User:Cscott/T206940&oldid=864158059

Hey, look at this:

$ (echo "[[File:Foo.jpg|caption|alt=''"; echo "alt'']]") | php maintenance/parse.php 
<p><a href="/~cananian/mediawiki/index.php/File:Foo.jpg" class="image" title="alt= alt"><img alt="alt= alt" src="/~cananian/mediawiki/images/0/06/Foo.jpg" width="400" height="267" /></a>
</p>

PHP doesn't handle this either. That is, it *is* an invalid alt, and *should* be treated as a caption AFAICT. However, it shouldn't break round-tripping...

And in particular it seems to be the newline between the double-quotes in the caption which is causing the alt attribute to fail.

$ (echo "[[File:Foo.jpg|caption|alt=''alt'']]") | bin/parse.js --normalize=parsoid

There's a misfeature in the PHP parser where "anything which doesn't properly parse as an option" is assumed to be a caption. Parsoid mimics this behavior, for better or for worse. (Predictably, see a new syntax proposal here.)

I've been using T90914 as the main task for this problem.

...and adding .jsonld at the end should also work, right?

I also worked on proper JSON+LD statements in T44063: [Epic] Provide a plain linked data interface for accessing entities and T164655: Store and serve annotations in W3C standard format. FWIW, with https://gerrit.wikimedia.org/r/384050 you get the following JSONLD for Q100:

{
    "@graph": [
        {
            "@id": "wdata:Q100",
            "@type": "schema:Dataset",
            "about": "wd:Q100",
            "license": "http://creativecommons.org/publicdomain/zero/1.0/",
            "softwareVersion": "0.1.0",
            "version": 2799,
            "dateModified": "2018-06-21T00:08:11Z",
            "statements": 85,
            "identifiers": 0,
            "sitelinks": 184
        },
        {
            "@id": "wd:Q100",
            "@type": "wikibase:Item"
        },
        {
            "@id": "https://sv.wikivoyage.org/wiki/Boston",
            "@type": "schema:Article",
            "about": "wd:Q100",
            "inLanguage": "sv",
            "isPartOf": "https://sv.wikivoyage.org/",
            "name": {
                "@language": "sv",
                "@value": "Boston"
            }
        },
  ...etc...
}

We should really have a 'Parsoid-Code-Debt' tag?

Well, I rebased my two patches on top of @Legoktm's patch and the builds are passing now, so I'll call that fixed.

Ah.

Thanks for the patch, looks good! And I'm fine with amending the original patch. I made a few minor suggestions. Thanks for testing this out in practice!

This bug is referenced from https://github.com/wikimedia/integration-config/blob/41e1e9a25d416e20ef4d236d5bff68c1e815538a/jjb/mediawiki.yaml#L265 :

# We do not run mediawiki/core tests with extensions installed
# https://phabricator.wikimedia.org/T197469#4293142

...but why?

In T205834#4631166, @matmarex wrote:

It is a bit interesting though that they are invisible, though. I would expect this issue to just make the variant annotations non-editable. Apparently Parsoid represents these nodes as empty <span> tags: https://sr.wikipedia.org/api/rest_v1/page/html/Papirus_1

<span typeof="mw:LanguageVariant" data-mw-variant="{&quot;disabled&quot;:{&quot;t&quot;:&quot;I&quot;}}" id="mwHQ"></span>

This was brought up again as desiderata due to caching/storage concerns. Since we don't actually provide data-parsoid to VE, we currently need to *guarantee* persistent storage of a matched set of Parsoid HTML/data-parsoid for that HTML for the entire duration of an editing session, to be certain that the html2wt phase can get back the appropriate data-parsoid.

I think I've fixed this! October's a little later than July but...

@Krenair SIGKILL is more reliable and would cost about the same as SIGHUP. Adding SIGHUP support would probably involve a lot of tedious tracking down various bits of cached data. Besides, much of this configuration management code is going away with the upcoming Parsoid/PHP port & integration with core.