Alternatively, could we just load monolog on-demand? I think there was an outstanding issue where we didn't actually want command-line invocations of parsoid to be logging to the production servers (or anywhere) by default. If we make no-logging (or log-to-stderr) the default command-line invocation, we wouldn't have to move monolog to prod dependencies. There should be no reason for mocks to be in the prod autoload, either. It would be nice if we could arrange things such that tools/ isn't needed: the entry point script can be in tools/ as long as it doesn't then depend on anything else in tools (at least not w/o manually adding that path to the autoload path).

More added in I250adfe741806400ffcb3e76b2a41b0a8b9422d8

In particular, switching to native HTML tags would be a huge dirty diff on a typical talk page, because you need to switch the tags both on the current list item but also all parent lists, and thus all their items and maybe subitems -- or else break the parent list and start a new list, which is a non-faithful serialization and would cause a WP:LISTGAP but which might still reasonably fall under the scrubWikitext functionality.

As a baseline at least, having extensions proactively declare which fundamental API tests they break at least gives us a way to tell exactly how "worrying" they are. Presumably the expectation would be that no extension would be enabled in production until/unless the set-of-tests-they-disable is zero.

Reproducing:

# production
$ curl -X GET --header 'Accept-Language: zh-cn' 'https://zh.wikipedia.org/api/rest_v1/page/html/User:Cscott%2FT241146/57725518'
{"type":"https://mediawiki.org/wiki/HyperSwitch/errors/unknown_error","method":"get","uri":"/zh.wikipedia.org/v1/page/html/User%3ACscott%2FT241146/57725518"}
# beta, after ssh to deployment-parsoid09.deployment-prep.eqiad.wmflabs
$ curl -H'Accept-Language: zh-cn' -x deployment-mediawiki-parsoid10:80 http://zh.wikipedia.beta.wmflabs.org/w/rest.php/zh.wikipedia.beta.wmflabs.org/v3/page/html/User:Cscott%2FZhTest/12308
# locally
$ php bin/parse.php --restURL "/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/User:Cscott%2FT241146" --htmlVariantLanguage zh-cn

It would be better to use ExtensionTemplateStyles to target .mw-parser-output to have this effect, rather than emit unbalanced tags in wikitext. Dunno if that's possible... but it should be.

An option might be to use T230683's <<< .... >>> syntax to handle T242777, and T230658's {{#attr}} to handle this task.

In T242466#5795063, @matmarex wrote:

I think Parsoid has workarounds in place for some other things that can't be represented in HTML, but clearly not this one. I don't know if that's some fundamental limitation or just something they didn't find important enough to spend time on.

Some discussion in T230683#5793251 and T230683#5798972, although that has more to do with general doBlockLevels brokenness and not necessarily <p> tag insertion in doBlockLevels. Eg, compare:

::<dl><dd>foo
bar</dd></dl>

with

**<dl><dd>foo
bar</dd></dl>

The results vary because doBlockLevels has no idea where it is in the list structure when it sees the newline after foo -- so it outputs some random tags, and then Remex has to clean the mess up the best it can.

In T230683#5798972, @Anomie wrote:

My point was that if <<< ... >>> is going to be a general-purpose "escaping" construct for terminators in wikitext, why doesn't it escape the </ref> too?

Tell me more about semantic attributes or accessibility attributes.

Extension:TemplateStyles is really a much better solution here. It will let you set up styles for ol, ol ol, ol ol ol, etc in whatever style alternation you prefer. The type attribute directly on list items is deprecated in HTML5, and inline style attributes are also being discouraged in wikitext in favor of proper encapsulation in a (scoped) stylesheet via TemplateStyles.

Note that these proposals add attributes to list *items* but not to the parent <ol>/<dl>/<ul> node. As discussed in T11996#5793656 there are various ways you can workaround this if you need to target the list container. Fundamentally the list container doesn't exist as wikitext syntax.

You may need to nest the heredoc syntax to get the effect you're looking for, if I understand correctly.

In T230683#5792972, @Anomie wrote:

In T230683#5791361, @cscott wrote:

"Just" changing the newline semantics of <div>

Note that's not what I proposed. There's nothing special about <div> there. The change in semantics is to SOL list markup, to "ignore" newlines embedded deeper in the DOM subtree when looking for the end of the list item.

@Anomie brings up some interesting corner cases over in T230683: New syntax for multiline list items / talk page comments for extensions of this syntax:

In T230683#5792972, @Anomie wrote:

Links, like [[Foo|<<< Some text with ]] in it >>>]]? Or [[What|<<<would [[this]] do?>>>]]?

In T11996#5792931, @Verdy_p wrote:

The clear intent in this example is to insert a floatting element inside a list item, not outside of it.

T230659#5787050 proposes using an {{#attr|id=....}} function in the (conceptual?) output of the signature parser function in order to assign an anchor to the comment. You might also consider adding a permalink, too -- using the proposal at the top of T234966 (probably out of date, but it will do for the purpose):

<span vocab="http://schema.org" typeof="Comment" class="mw-signature">
    <a href="/wiki/User:cscott" property="creator">C. Scott Ananian</a>
    <a href="#<generated id>">  <!-- THIS IS THE PERMALINK BIT -->
    <time property="dateCreated" datetime="2007-03-29T18:07Z">18:07, 29 March 2007 (UTC)</time>
    </a>
</span>

(Note that the actual id attribute matching that would be added to the list item holding the comment, the *parent node* of the signature, using the {{#attr}} mechanism.)

In T230683#5791184, @Anomie wrote:

*** <div>Text text
Text text</div>

work like it already does with an opaque extension tag like <syntaxhighlight>.

To some degree "HTML markup on talk pages" and "wikitext markup on talk pages" are orthogonal. If we can agree on what "proper" wikitext looks like, then we can certainly tweak the parser so that it outputs different semantic tags for that markup when in the talk namespace. That's oversimplifying a bit -- a bunch of downstream stuff would need to know about the output change as well (VE, gadgets, CSS, etc) -- but all that downstream stuff would also need to change if we added new markup specifically for talk page replies. So conceptually we can probably separate the concerns.

(As such, this is a dup of T241146.)

Fixed by https://gerrit.wikimedia.org/r/559493 -- which was supposed to be in the "next deploy" but looks like I missed the boat somehow.

In T204366#5787356, @Anomie wrote:

In T204366#5787301, @cscott wrote:

But # isn't a valid numeric or named parameter, so it can be used as an unambiguous prefix for the new varargs functionality.

That's incorrect, it's perfectly valid as a named parameter.

TL;DR: you're right, I should probably have proposed "{{#attr}} doesn't escape templates to affect content in the surrounding page" as the basecase, although you could support that (keep reading...)

Syntax comparison from the meeting notes:

"Wikitext 2.0" {: ... :}

• Example

:::{: line 1

I was deliberately trying to downplay "consensus" with "rough" and suggest only that the winds were vaguely blowing in that direction. But it's clear the words I chose read differently to you than they did in my head. I apologize.

Implicit ending of list items by newlines is probably too wide-spread in existing wikitext to change now.
But T230683: New syntax for multiline list items / talk page comments and the end of T114432: [RFC] Heredoc arguments for templates (aka "hygienic" or "long" arguments) discuss ways to escape the newline where desired.

The "rough consensus" in T230683#5787066 the result of a meeting for that purpose involving @cscott, @ssastry, @Whatamidoing-WMF, @dchan, @matmarex, @ppelberg, Jazmin, @Esanders, @DLynch, and Marcella. (I don't personally have a strong preference among the alternatives suggested and didn't vote.) That meeting obviously doesn't include all possible stakeholders, which is why I posted here to try to keep everyone in sync. As I understand it, the goal is to put forward an RfC with one of these proposed syntaxes, which would of course get further debated as usual for that process.

In T204366#5782724, @Anomie wrote:

Indeed. The general idea is better than JSON for wikitext, why not lose the JSON entirely?
It wouldn't surprise me if someone already has done this somewhere, having the equivalent of the {{item}} template output a delimited record and parsing that record format in a module.

Rough consensus seems to be forming around:

:::: <<< line 1

As a concrete proposal, I'd like to tie this in with the strawman in T230658#5786980 and propose that the magic parser function {{#~}} (see T230653: Use a parser function to encapsulate signatures) "conceptually" include {{#attr|id=<something>}} in its expansion. (See the discussion there about p-wrapping.)

To narrow down the options some, I'd like to put forward

::: {{#attr|id=bar|class=x}}

as a general "attach these attributes to the containing HTML tag" mechanism. It would work in templates, so you could localize it/come up with useful shortcuts.

In T230658#5549354, @Izno wrote:

Syntax for attributes is something I made a task for at T202083: First-class wikitext support for ordered list item value which I closed duplicate of the problem I was looking to solve in that context. General gist I suggested was similar to the table syntax, namely: # id=A class=X | BCD where BCD is the list item content. I can see users going for that syntax. I don't really understand the proposed "table" syntax above (:::|id=foo|class=bar| ("like table syntax")) since that's not how table syntax works. "like", but definitely not.

Some additional comments:

Best description of this proposal so far is in the talk; slides start here:
https://commons.wikimedia.org/w/index.php?title=File:Wikitext_2.0.wikimedia.devsummit.2017.pdf&page=56

From david: “The ability to enter :{| into any page and have it strip all newlines the next time Parsoid interacts with said page” isn’t great!

I *think* this is because wikitech doesn't use RESTBase and talks to Parsoid directly... and I think *that's* because wikitech is on a different cluster and can't physically talk to RESTBase (and now Parsoid)? Can someone confirm that? I *think* this is more of an ops problem than a Parsoid bug...

Reproducible with I02c7fdd34d6808497a6aebf2d35e7073f49a0286:

$ php bin/parse.php --restURL "/w/rest.php/uk.wikipedia.org/v3/page/pagebundle/%D0%9C%D0%BE%D0%BD%D1%83%D0%BC%D0%B5%D0%BD%D1%82/26794587"
Wikimedia\Assert\InvariantException from line 217 of /home/cananian/Projects/Wikimedia/Parsoid/vendor/wikimedia/assert/src/Assert.php: Invariant failed: Bad UTF-8 at end of string (2 byte sequence)
#0 /home/cananian/Projects/Wikimedia/Parsoid/src/Utils/PHPUtils.php(238): Wikimedia\Assert\Assert::invariant(false, 'Bad UTF-8 at en...')
#1 /home/cananian/Projects/Wikimedia/Parsoid/src/Tokens/SourceRange.php(82): Parsoid\Utils\PHPUtils::safeSubstr(': {{otheruses|\xD0...', 410, 5)
#2 /home/cananian/Projects/Wikimedia/Parsoid/src/Tokens/Token.php(254): Parsoid\Tokens\SourceRange->substr(': {{otheruses|\xD0...')
#3 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/Sanitizer.php(1292): Parsoid\Tokens\Token->getWTSource(Object(Parsoid\Wt2Html\PageConfigFrame))
#4 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/Sanitizer.php(1514): Parsoid\Wt2Html\TT\Sanitizer::sanitizeToken(Object(Parsoid\Config\Env), Object(Parsoid\Tokens\TagTk), false)
#5 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/TokenHandler.php(239): Parsoid\Wt2Html\TT\Sanitizer->onAny(Object(Parsoid\Tokens\TagTk))
#6 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(120): Parsoid\Wt2Html\TT\TokenHandler->process(Array)
#7 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(182): Parsoid\Wt2Html\TokenTransformManager->processChunk(Array)
#8 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/ParserPipeline.php(127): Parsoid\Wt2Html\TokenTransformManager->process(Array, Array)
#9 /home/cananian/Projects/Wikimedia/Parsoid/src/Utils/PipelineUtils.php(110): Parsoid\Wt2Html\ParserPipeline->parse(Array, Array)
#10 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/DOMFragmentBuilder.php(99): Parsoid\Utils\PipelineUtils::processContentInPipeline(Object(Parsoid\Config\Env), Object(Parsoid\Wt2Html\Frame), Array, Array)
#11 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/DOMFragmentBuilder.php(122): Parsoid\Wt2Html\TT\DOMFragmentBuilder->buildDOMFragment(Object(Parsoid\Tokens\SelfclosingTagTk))
#12 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/TokenHandler.php(211): Parsoid\Wt2Html\TT\DOMFragmentBuilder->onTag(Object(Parsoid\Tokens\SelfclosingTagTk))
#13 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(120): Parsoid\Wt2Html\TT\TokenHandler->process(Array)
#14 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(182): Parsoid\Wt2Html\TokenTransformManager->processChunk(Array)
#15 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/ParserPipeline.php(127): Parsoid\Wt2Html\TokenTransformManager->process(Array, Array)
#16 /home/cananian/Projects/Wikimedia/Parsoid/src/Utils/PipelineUtils.php(110): Parsoid\Wt2Html\ParserPipeline->parse('[[\xD0\xA4\xD0\xB0\xD0\xB9\xD0\xBB:\xD0\x9F\xD0\xB0...', Array)
#17 /home/cananian/Projects/Wikimedia/Parsoid/src/Config/ParsoidExtensionAPI.php(158): Parsoid\Utils\PipelineUtils::processContentInPipeline(Object(Parsoid\Config\Env), Object(Parsoid\Wt2Html\Frame), '[[\xD0\xA4\xD0\xB0\xD0\xB9\xD0\xBB:\xD0\x9F\xD0\xB0...', Array)
#18 /home/cananian/Projects/Wikimedia/Parsoid/src/Ext/Gallery/Gallery.php(181): Parsoid\Config\ParsoidExtensionAPI->parseWikitextToDOM('[[\xD0\xA4\xD0\xB0\xD0\xB9\xD0\xBB:\xD0\x9F\xD0\xB0...', Array, true)
#19 /home/cananian/Projects/Wikimedia/Parsoid/src/Ext/Gallery/Gallery.php(276): Parsoid\Ext\Gallery\Gallery::pLine(Object(Parsoid\Config\ParsoidExtensionAPI), '\xD0\xA4\xD0\xB0\xD0\xB9\xD0\xBB:\xD0\x9F\xD0\xB0\xD0\xBC...', 11753, Object(Parsoid\Ext\Gallery\Opts))
#20 [internal function]: Parsoid\Ext\Gallery\Gallery->Parsoid\Ext\Gallery\{closure}(Array)
#21 /home/cananian/Projects/Wikimedia/Parsoid/src/Ext/Gallery/Gallery.php(274): array_map(Object(Closure), Array)
#22 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/ExtensionHandler.php(132): Parsoid\Ext\Gallery\Gallery->toDOM(Object(Parsoid\Config\ParsoidExtensionAPI), '\n\xD0\xA4\xD0\xB0\xD0\xB9\xD0\xBB:Denkm...', Array)
#23 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/ExtensionHandler.php(258): Parsoid\Wt2Html\TT\ExtensionHandler->onExtension(Object(Parsoid\Tokens\SelfclosingTagTk))
#24 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TT/TokenHandler.php(211): Parsoid\Wt2Html\TT\ExtensionHandler->onTag(Object(Parsoid\Tokens\SelfclosingTagTk))
#25 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(120): Parsoid\Wt2Html\TT\TokenHandler->process(Array)
#26 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(192): Parsoid\Wt2Html\TokenTransformManager->processChunk(Array)
#27 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/TokenTransformManager.php(190): Parsoid\Wt2Html\TokenTransformManager->processChunkily(': {{otheruses|\xD0...', Array)
#28 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/HTML5TreeBuilder.php(430): Parsoid\Wt2Html\TokenTransformManager->processChunkily(': {{otheruses|\xD0...', Array)
#29 [internal function]: Parsoid\Wt2Html\HTML5TreeBuilder->processChunkily(': {{otheruses|\xD0...', Array)
#30 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/DOMPostProcessor.php(894): Generator->current()
#31 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/ParserPipeline.php(148): Parsoid\Wt2Html\DOMPostProcessor->processChunkily(': {{otheruses|\xD0...', Array)
#32 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/ParserPipeline.php(198): Parsoid\Wt2Html\ParserPipeline->parseChunkily(': {{otheruses|\xD0...', Array)
#33 /home/cananian/Projects/Wikimedia/Parsoid/src/Wt2Html/ParserPipelineFactory.php(299): Parsoid\Wt2Html\ParserPipeline->parseToplevelDoc(': {{otheruses|\xD0...', Array)
#34 /home/cananian/Projects/Wikimedia/Parsoid/src/WikitextContentModelHandler.php(78): Parsoid\Wt2Html\ParserPipelineFactory->parse(': {{otheruses|\xD0...')
#35 /home/cananian/Projects/Wikimedia/Parsoid/src/Parsoid.php(93): Parsoid\WikitextContentModelHandler->toHTML(Object(Parsoid\Config\Env))
#36 /home/cananian/Projects/Wikimedia/Parsoid/src/Parsoid.php(123): Parsoid\Parsoid->parseWikitext(Object(Parsoid\Config\Api\PageConfig), Array)
#37 /home/cananian/Projects/Wikimedia/Parsoid/bin/parse.php(255): Parsoid\Parsoid->wikitext2html(Object(Parsoid\Config\Api\PageConfig), Array)
#38 /home/cananian/Projects/Wikimedia/Parsoid/bin/parse.php(460): Parse->wt2Html(Array, Array, NULL)
#39 /home/cananian/Projects/Wikimedia/Parsoid/tools/doMaintenance.php(53): Parse->execute()
#40 /home/cananian/Projects/Wikimedia/Parsoid/bin/parse.php(476): require_once('/home/cananian/...')
#41 {main}

Fixed in langconv 0.3.2 (commit 5ea2fe04fe4ace60c7b6f0fc96c77dc4e38c640f).

I suspect a more forward-compatible hook would be to hook link resolution, and pass in a reference to the frame there. The legacy parser is being replaced by Parsoid, so hooks based on the legacy parser aren't going to be maintainable long-term. However, Semantic MediaWiki wants to customize how [[....]] syntax is resolved, so we are likely to maintain some sort of hook based around that syntax.

Fixed in domino 2.1.4.

That may be a good idea.

In T240057#5721283, @Anomalocaris wrote:

When wikitext linting updates are re-enabled, what will happen to everything that was edited while wikitext linting updates was disabled? How long will it take to get caught up?

PCS load is still higher than before Nov 19. Hopefully it will return to baseline when we turn off language converter mirroring to Parsoid/JS, although in theory that should be a very small # of requests/second. May still be worth looking into why the base traffic stayed high.

Probably related to {T239983} and the general API server unhappiness in that time period. Seems to have resolved itself when we fixed T239983 (bought ourselves a year at least) by turning off Parsoid/JS linting to shed API server load.

What's the plan for rolling the train forward now? Are we just skipping last week's train entirely, or are we going to try to roll everyone to wmf.8 today (Monday)?

Seems like this has caused T239877 because of a matching but opposite error in the query planner. My guess is that WHERE foo=1 skips the index for foo if foo is a string type, perhaps because it wants to convert each foo string to an integer individually to account for 01 and -0 and similar isomorphisms.

I bet that WHERE foo=1 bypasses the index and does a scan of every single title string and converts it to an integer to cover the cases where the title might be (say) 01 or 1., whereas WHERE foo='1' uses the db index as you'd expect.

So VE bug, in that it should strip data-parsoid during paste?

return $start <= $end ? substr( $this->selserData->oldText, $start, $end - $start ) : null;

and then:

If string is less than start characters long, FALSE will be returned.
If length is given and is negative, then that many characters will be omitted from the end of string (after the start position has been calculated when a start is negative). If start denotes the position of this truncation or beyond, FALSE will be returned.

It's an invalid UTF-8 string, which is then causing the FST to fail (because it doesn't have edges corresponding to invalid UTF-8 sequences). The string in stack trace #2 there looks like a bogus fragment -- that's some text from the middle of -{ }- markup but the boundaries don't look right to me -- I'm guessing that's the underlying problem.

Ah! Can now reproduce:

$php bin/parse.php --restURL '/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/%E8%89%BE%E5%A1%9E%E5%88%97%E5%85%AC%E7%88%B5' --htmlVariantLanguage zh-hans
Wikimedia\Assert\InvariantException from line 217 of /home/cananian/Projects/Wikimedia/Parsoid/vendor/wikimedia/assert/src/Assert.php: Invariant failed: /home/cananian/Projects/Wikimedia/langconv/src/../fst/brack-zh-hant-noop.pfst
#0 /home/cananian/Projects/Wikimedia/langconv/src/FST.php(149): Wikimedia\Assert\Assert::invariant(false, '/home/cananian/...')
#1 /home/cananian/Projects/Wikimedia/langconv/src/FST.php(208): Wikimedia\LangConv\FST->Wikimedia\LangConv\{closure}()
#2 /home/cananian/Projects/Wikimedia/langconv/src/ReplacementMachine.php(92): Wikimedia\LangConv\FST->run('\xE7\xBE\x85\xE6\xAF\x94; zh-hk:\xE7...', 0, 46, true)
#3 /home/cananian/Projects/Wikimedia/langconv/src/ReplacementMachine.php(187): Wikimedia\LangConv\ReplacementMachine->countBrackets('\xE7\xBE\x85\xE6\xAF\x94; zh-hk:\xE7...', 'zh-hant', 'zh-hant')
[...]

It's not a regression, see above.

Hitting
https://ar.wikipedia.org/api/rest_v1/page/html/%D8%A8%D9%88%D8%A7%D8%A8%D8%A9%3A%D8%B9%D9%84%D9%85_%D8%A7%D9%84%D9%88%D8%B1%D8%A7%D8%AB%D8%A9%2F%D9%85%D8%AC%D8%A7%D9%84_%D9%85%D8%AE%D8%AA%D8%A7%D8%B1?redirect=false
directly gives a 500 error, though, and it doesn't take too long (so probably not a timeout). So something definitely wrong somewhere...

Hm, selser looks fine:

$ (echo 'foo' ; echo '' ; echo '<gallery>' ; echo 'Foo.jpg|foo' ; echo '</gallery>') > T239783.wt
$ php bin/parse.php --wt2html < T239783.wt > T239783.html
$ sed -e 's/>foo<\/p>/>bar<\/p>/' < T239783.html > T239783.2.html
$ php bin/parse.php --selser --oldtextfile T239783.wt --oldhtmlfile T239783.html < T239783.2.html
bar

Reproducible:

$ (echo '<gallery>' ; echo 'Foo.jpg|foo' ; echo '</gallery>') | php bin/parse.php --wt2wt
<gallery>
File:Foo.jpg|foo
</gallery>

But this behavior is in Parsoid/JS as well:

$ (echo '<gallery>' ; echo 'Foo.jpg|foo' ; echo '</gallery>') | bin/parse.js --wt2wt
<gallery>
File:Foo.jpg|foo
</gallery>

And this behavior was present in Parsoid/JS at least as far back as Oct 29 / 42f33caad4f954a3cdea09301472e42bc8478ce2.

Can't immediately reproduce this on the command line in API mode, eg this produces no errors:

$ php bin/parse.php --restURL 'https://ar.wikipedia.org/api/rest_v1/page/html/%D8%A8%D9%88%D8%A7%D8%A8%D8%A9%3A%D8%B9%D9%84%D9%85_%D8%A7%D9%84%D9%88%D8%B1%D8%A7%D8%AB%D8%A9%2F%D8%A5%D8%B6%D8%A7%D8%A1%D8%A9_%D8%B9%D9%84%D9%89?redirect=false'

Maybe something related to the integrated mode, or a resource limit of some kind?

Well, one reason I'm having trouble reproducing this is that the logged errors don't (apparently) contain the actual page title being processed; see T239768: All RestBase mirrored html2html language conversion pages have page title set to Main Page.

It's a stack underflow; I can confirm that PHP emits warnings instead of crashing when you dereference null here (weird!).

In T239643#5708833, @ssastry wrote:

In T239643#5707064, @cscott wrote:

I believe these are the result of BacktrackState being omitted from the repo (https://gerrit.wikimedia.org/r/554085):

No, these errors still exist. See logstash from this morning after Marko mirrored traffic. These are all still present in large volumes.

I managed to create
https://en.wikipedia.org/w/index.php?title=42_Test_for_T239666&action=history
with the new wikitext editor, and
https://en.wikipedia.org/w/index.php?title=42_Test_for_T239666_2&action=history
with visual editor and didn't see any problems.

(Might actually be a bug in core w/ integer titles? https://gerrit.wikimedia.org/r/554122 was failing parsertests in core, not in parsoid...)

Tagging RESTBase as well, just in case it's a bug in the middle layer.

These should be handled by https://gerrit.wikimedia.org/r/554113:

PHP Notice: Undefined property: stdClass::$disabled
PHP Notice: Undefined property: stdClass::$oneway
PHP Notice: Undefined property: stdClass::$twoway
PHP Notice: Undefined property: stdClass::$name

I believe these are the result of BacktrackState being omitted from the repo (https://gerrit.wikimedia.org/r/554085):

PHP Notice: Trying to get property 'epsEdge' of non-object
PHP Notice: Trying to get property 'blen' of non-object
PHP Notice: Trying to get property 'outpos' of non-object
PHP Notice: Trying to get property 'idx' of non-object

The <templatestyles> extension appears to be implicated as well.

Minimal repro case:

<includeonly>{| border='1'
|}
</includeonly><noinclude>
{{Documentation|contenu=
== Exemple ==

Reproducible!

$ tools/fetch-wt.js --domain fr.wikipedia.org 164858496 > T239252.wt
$ php bin/parse.php --pageName 'Utilisateur:Warp3/test4' --domain fr.wikipedia.org < T239252.wt
[warn/dsr/inconsistent] DSR inconsistency: cs/s mismatch for node: span s:4830 ; cs:4829
[warn/dsr/inconsistent] DSR inconsistency: cs/s mismatch for node: span s:3799 ; cs:3798
Wikimedia\Assert\InvariantException from line 217 of /home/cananian/Projects/Wikimedia/Parsoid/vendor/wikimedia/assert/src/Assert.php: Invariant failed: Bad SourceRange length

Hm, another one I can't seem to reproduce:

$ tools/fetch-wt.js --domain fr.wikipedia.org 139698506 > T236866.wt
cananian@skiffserv:~/Projects/Wikimedia/Parsoid$ php bin/parse.php --pageName 'Modèle:Hiérarchie fin nénètse' --domain fr.wikipedia.org --offsetType ucs2 < T236866.wt 

No problems. But this is another page which is mostly a template inclusion; highly likely the triggering template changed.

Can't reproduce with the latest version of that page or with revision id 56190196:

$ php bin/parse.php --pageName 'Wikipedia:Biểu_quyết_đánh_giá_bài_viết_dịch_thuật' --domain vi.wikipedia.org < /dev/null
$ tools/fetch-wt.js --domain vi.wikipedia.org 56190196 > T237318.wt
$ php bin/parse.php --pageName 'Wikipedia:Biểu_quyết_đánh_giá_bài_viết_dịch_thuật' --domain vi.wikipedia.org < T237318.wt

Perhaps a problem with a template, which has since been updated? Proper template time travel would really be useful.

This should be mostly addressed by I58d54dec51e8a4da0d1f0ba68144a2dd0cac6235 / 9d17caeca8310a748c956a6c3f8efbf3c9073d49 in the langconv library. That still does a file_get_contents() to load the FST; we could compile these into PHP files with a string constant to allow caching the data across PHP instances. Waiting for initial performance metrics to determine if that additional step is necessary (or effective: it may be that APC refuses to cache such large class files).

Seems to be present on both enwiki and officewiki, so not a Parsoid/PHP issue.

officewiki is running 1.35.0-wmf.5 (a473ba0) and VisualEditor 0.1.1 (26ebdd0) 14:26, 5 November 2019.

To elaborate a little bit, our intention is to tie a bow around production deployment of Parsoid/PHP in the next month or so, then January (roughly) after we've retired Parsoid/JS start looking at code debt and integration issues with the Parsoid/PHP codebase. Moving away from PHP's dom extension is maybe not #1 on the list of things to do in that phase, but probably in the top 5 at least.

So, if I'm understanding correctly, RESTBase is always going to add the header on (say) zhwiki, but the usual sequence is: