Parsoid uses <section>, <figcaption>, and <figure> already in its output (and thus the main parser will too, as Parsoid is merged into core). <picture> could be considered as part of media layout, but we are using other better-supported mechanisms for responsive images at the moment. These tags should not be whitelisted in article content as they conflict with wikitext features.

I think HTML5 says that an empty attribute is actually itself, not the empty string? (Let me check...)

These sort of patches should be ported to Parsoid / at least have Parsoid ping'ed regarding them.

If you're using Visual Editor, it would auto-complete the proper ISBN markup without anyone having to learn it specifically.

I'm inclined to let this age out. In addition to being based on the OCG framework and the old VM infrastructure, it's based on the Parsoid/JS service, which is also due to be replaced in the new year. It would be a good April 1, 2019 project to resurrect it based on Parsoid/PHP (and no OCG), perhaps.

Thanks, all. Step by step...

If you could provide more details, I'd certainly be interested in helping debug the XPath library interaction. Domino is pretty heavily performance-optimized at this point.

I think this bug is fixed everywhere now. Resolving; reopen if you find a problem that can't be fixed by purging parsercache.

Yeah, I need to port https://gerrit.wikimedia.org/r/445664 to Parsoid. The BCP 47 codes are exported from siteinfo as of https://gerrit.wikimedia.org/r/460038 which was merged in October, so if we haven't sync'ed the baseconfigs since then we probably need to.

Starting points, copied from IRC:

Yes, true. But we need some way to disambiguate between returning \xFF meaning the literal byte 255 and \u00FF meaning the UTF-8 byte sequence denoting codepoint 255.

I don't have any problem with using the unicode codepoint. I just don't think we should invent a bogus entity name for it. You can use the codepoint as a hex or decimal character reference in wikitext. Perhaps even file a bug with the W3C/WHATWG to add the &cgj; entity upstream. Many places in MW assume that the MW entity names correspond to valid HTML entity names; I don't think wikitext shouldn't have its own nonstandard entities.

https://en.wikipedia.org/wiki/User:Cscott/T209236 is my test case, but I agree: even after purging the page I'm still seeing a &para; on the figure link. But that makes sense 'cuz you rolled it back...

Probably the most straightforward solution would be to deprecate the "properties" value in ApiParse.php and replace it with a "binProperties" value which is the bin2hex'ed value of each property. That would ensure that there was some means of exporting even binary page properties, such as those used by Kartographer, TemplateData, Graph, etc.

As an example, PHP var_export says the raw value is:

'kartographer' => ' � ' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . ' E�� �0 E�e�HJKm� \\�7��1 "P2� ���e�nϹ��pW�h!�uZ��y�E�}� 3��ʣc�83!� ���}�GC �S �@ 3&�g�R�>̼���' . "\0" . '' . "\0" . '' . "\0" . '',

but the value exported by the API for this is:

"kartographer": "\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdE\ufffd\ufffd\ufffd\ufffd0\ufffdE\ufffde\ufffdHJKm\ufffd\ufffd\\\ufffd7\ufffd\ufffd1\ufffd\"P2\ufffd\ufffd\ufffd\ufffd\ufffde\ufffdn\u03f9\ufffd\ufffdpW\ufffdh!\ufffduZ\ufffd\ufffdy\ufffdE\ufffd}\ufffd\ufffd3\ufffd\ufffd\u02a3\u436e\ufffdU\ufffd\ufffdv\ufffd\u06c2\ufffd\ufffd\ufffdM|\ufffdB\ufffd\ufffd\ufffd\ufffdF\u06a0\ufffd\ufffdk\ufffd\u6615\ufffd\ufffd\ufffd0\ufffd\ufffdk\ufffd\ufffdR\ufffdJ\ufffd\ufffdY\ufffd\ufffdn\ufffd\ufffdR\ufffd\ufffd\ufffd\ufffd)\ufffd\ufffd`\ufffd\ufffd\ufffd\ufffd\ufffdn\ufffd\rc\ufffd83!\ufffd\ufffd\ufffd\ufffd}\ufffdGC\ufffd\ufffdS\ufffd\ufffd@\ufffd3&\ufffdg\ufffdR\ufffd>\u033c\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd",

Note that all the \0 have become \ufffd.

Script to generate regexp matching all semicolon-less HTML entities is at P7844; this was used in the patch linked above.

Confirmed the bug exists in core (but not in <gallery>) and in Parsoid (both in native links and <gallery>). &para[A-Za-z0-9=] should never be entity decoded, according to https://www.w3.org/TR/html5/syntax.html#character-reference-state ; investigating what's going on here.

Let me take a quick look. Apologies to German WP, this was a long holiday weekend in the US which probably delayed attention to this problem.

Well, I added you as co-owner of the jsdoc-wmf-theme group. That should do it, I hope?

Fine with me! I'm the owner of the group, how do I go about giving you +2 rights I wonder?

I need to dig into PHP semantics. Is it possible you might be suppressing the notice on $parent->children because of the =& instead of plain =? (line 246)

Did you check the other issues from https://phabricator.wikimedia.org/T206940#4670526 ? Sounds like you're saying that wt2html is fixed (both PHP and Parsoid agree and do something sensible) but that html2wt for video specifically is still broken since it treats the (invisible) embedded alt attribute as plaintext rather than HTML. Or maybe html2wt is alright but we shouldn't be embedding the invisible alt as HTML but should be doing the same tag-stripping that we would do for a "real" alt attribute. (I think I'm a little partial to this latter.)

Copied from a discussion at https://gerrit.wikimedia.org/r/#/c/mediawiki/services/parsoid/+/467531/6/lib/wt2html/tt/LinkHandler.js@1014 wrt how [[File:Foo.jpg|{{sometemplate}}]] gets parsed:

In T208549#4716336, @Joe wrote:

• I didn't check the size of the bytecode cache databases to see if that exploded. If that's the case, I might want to purge it today.

A bunch of new messages are getting tidied and going through remex now ( https://gerrit.wikimedia.org/r/#/q/topic:deprecate-wgtidy ) -- although that *shouldn't* cause problems, here are a couple of wild theories:

1. MessageCache is missing somehow ( 4b1db1190bb8f2a115c6a81a5ee487b7d18cd303 ) and tying up CPU (but I'm pretty sure the misses would show in logs). Rolling back would cause misses on the updated messages so "not solve the issue".
2. ParserCache is missing somehow ( 58abac2d1489cdfaaf2ffdf2f9e1214509760b31 ) but I'm pretty sure that would show up in analytics. Same thing, roll back could cause misses of the previous misses.
3. Some system message is triggering some weird infinite-loop bug in Remex. Of course, remex has been used for tidy in all article content for months now, but I'm always ready to be surprised. If there's a specific URL which triggers the CPU hog behavior, that would be a big clue. But wouldn't explain why rolling back didn't help.
4. It's not my fault at all. (I like this one the best, but it doesn't help you fine folks in ops any.) ;)

SEO optimization came up on the Audiences 1 QCI presentation, and it was mentioned that one question we had was whether Google used the same ingestion pipeline for all languages / wikis, or whether there were certain things that would work differently on (say) English wikipedia -vs- Spanish wikisource.

I think I could be satisfied with a good documentation comment for this method elaborating these points:
(a) state explicitly that this uses the content language, not the user interface language
(b) is intended for parser functions and tag hooks which appear in article content, not UX (special pages, warnings, etc)
(c) the result will be subject to language conversion (which is unusual for system messages, which are more usually pre-converted to a specific variant)

See also T114640: RFC: make Parser::getTargetLanguage aware of multilingual wikis, which goes in to more detail about the problems one has in general when trying to make "user interface elements" from "content" markup.

I'm not convinced the helper should be in Parser.php. Parser functions and tag hooks are a bit of a special case, since they actually are in the content language. In 95% of the cases where system messages are used, they should be in the user interface language -- compare https://codesearch.wmflabs.org/search/?q=addWikiTextAsContent with https://codesearch.wmflabs.org/search/?q=addWikiTextAsInterface for example. Even when you do want the content language, it's probably because the thing you're adding would be better off as an interface message, for example see bacd87e4942baa34808a1b77d3b29bfdb566cc17.

Verified that rETRAb2586aebd94d: Avoid untidy calls to OutputPage::addWikiText() didn't make it to the REL1_32 branch of Translate, and so no further backporting should be needed.

I think you still need to sanitize -- for example, the <script> tag has its payload delivered by the node children, not the attributes. Potentially <style> is dangerous in the same way, if there is a CSS vulnerability. But again, a tag whitelist should be sufficient for this.

Patch at https://github.com/wikimedia/language-data/pull/37

w00t! Now we can do https://gerrit.wikimedia.org/r/368248 (T117845) ?

There's a lot of backlog to read through but -- yeah, Wikimedia has *always* used their own patched version of tidy. The stock debian libtidy has never exactly matched the WMF version, although it's possible the precise differences weren't previously well-covered by parserTests. All use of libtidy is deprecated and is being removed ( https://gerrit.wikimedia.org/r/467972 ) so it's just a matter of keeping our tests of long-term-supported releases sane by maintaining access to the WMF-patched-version-of-old-tidy.

Thanks, @Legoktm!

OK, two patches to fix the issue: belt and suspenders. Both of them are potential candidates for cherry-picking to 1.32, but let's get them merged on 1.33 first.

I bet I technically have all the required permission bits to do this myself -- looks like @Legoktm made the last release -- but it would be helpful to have some instructions, either on-wiki or in the README, since I've never published a composer package before.

Ah, ok, thanks! That means I don't have to worry about this being an "unbreak now" sort of bug.

Arrrrgh. WikiBase/lib/includes/LanguageWithConversion.php contains code cut-and-pasted from mediawiki-core. Anyone want to guess the odds that it wasn't updated when the code from core was updated?

Ok, merged T207447: uselang=zh-hant-hk causes fatal exception of type "BadMethodCallException" into this one, because the stack trace is pretty much identical:

Also -- sr-cyrl, zh-hans-tw, etc are not actually the mediawiki-internal names for these languages. https://gerrit.wikimedia.org/r/460039 would have added support for using the standard names, but I'm a little bit surprised that anything is generating links to these "non-mediawiki" (but BCP 47 standard) codes. I mean, it's good -- we *should* be trying to move to using the proper BCP 47 codes -- but it's worth trying to track down where these links are coming from, since they wouldn't have worked prior to 460039.

Given that uselang processing is involved, perhaps https://gerrit.wikimedia.org/r/460039 is the proximate cause. Can we get a stack trace for that exception?

Re question #3 wrt JS-vs-PHP: it's important to understand the role of the social ecosystem involved. One reason why markdown is widespread (and wikitext is not used at all outside the WMF environment) is that we've never had a really good/fast standalone parser. Even our own WMF research department uses mwparserfromhell (written in Python) instead of either of the two "official" WMF parsers. (And they probably won't migrate to an official parser even once/if Parsoid is in PHP.)

Re the first question, I feel the "significance" section presupposes a particular answer (ie, that WYSIWYG editors aren't appropriate for templates, and so wikitext editing should be limited). I don't agree (T114454), but regardless, here's my attempt at rephrasing the prompt in a neutral manner:

Any updates on this task? As I said, I'd like to write Scribunto/JS using v8js at some point. If v8js would be helpful to wikibase, perhaps we can pool efforts.

It's not happening in 1.32, we just branched that. Hopefully for 1.33!

I filed T207297: Phan SecurityCheck-XSS and SecurityCheck-SQLInjection errors in SecurePoll extension out of an abundance of caution, but it's probably a dup. Slightly different errors, though...

@Osnard Could you see if my updated patchset works for you? Rather than explicitly test reverseMwApiMap again, I just unconditionally called ParsoidConfig#getPrefixFor() to set the prefix (even if it was already set). That's a little more consistent with the direction we really want to go (T206764: Remove `prefix` from Parsoid and use `domain` consistently as configuration key).

In https://gerrit.wikimedia.org/r/446766 I introduced BabelLanguageCodes::getCategoryCode() which maps mediawiki-internal language codes to appropriate category names. The current algorithm is to use the (lowercased) mediawiki internal code if it doesn't contain a hyphen (eg en, simple, de), otherwise use the properly-capitalized BCP 47 code (zh-Hans, etc). This matched previous expectations as canonized in the extensions phpunit tests. If we wanted some other behavior for category codes it ought to be straightforward to patch getCategoryCode() for whatever is desired.

The patch is merged, but we're going to need a new version of remex released to composer and mediawiki-core updated to require the new version before this bug is actually fixed in production uses of php 7 (ie, when running the php 7 jenkins tests).

We only emit the "@graph" form in purtle if the API is used to annotate more than a single entity: https://github.com/wikimedia/purtle/blob/master/src/JsonLdRdfWriter.php#L37

In T189966#4323531, @tstarling wrote:

• Special CDN integration with the Key header (formerly X-Vary-Options). X-Vary-Options was a feature I introduced to avoid splitting the CDN cache between IE and everything else, which were sending slightly different Accept-Encoding request headers (differing by a space)

This turned into a little rathole, but I've come out the other end fixing (a) how Parsoid parses alt/link options (wikitext markup including <nowiki> is allowed), (b) how Parsoid renders alt/link options (consistent stripping), (c) how core renders link options (<nowiki> expansion and stripping consistent with alt) , (d) how core handles ampersands in alt/link options (bug in remex), and (e) how Parsoid handles ampersands in alt/link options. Now we've just got to get those three patches merged, starting with the remex bug (T207088: Remex double-decodes HTML entities on PHP (not HHVM)) because the newly-added test cases won't pass on jenkins until remex is fixed and the fix is packaged and released so composer can get to it.