@mobrovac, Can I punt this puppet updates to you? We cannot do this via the horizon UI and this needs a puppet patch. I tried enabling role::parsoid::testing via the UI and it needs a parsoid_port hieradata property which isn't available right now.

So, y'day I dumped the regexp on scandium on one of the test pages and here it is:

'/^((?:(?:#REDIRECT)[ \t\n\r\x0c]*(?::[ \t\n\r\x0c]*)?\[\[[^\]]+\]\])?(?:\[\[Category\:[^\]]*?\]\]|__(?:(?i:(?P<a_notoc>__NOTOC__)|(?P<a_nogallery>__NOGALLERY__)|(?P<a_forcetoc>__FORCETOC__)|(?P<a_toc>__TOC__)|(?P<a_noeditsection>__NOEDITSECTION__)|(?P<a_notitleconvert>__NOTITLECONVERT__)|(?P<b_notitleconvert>__NOTC__)|(?P<a_nocontentconvert>__NOCONTENTCONVERT__)|(?P<b_nocontentconvert>__NOCC__))|(?P<a_newsectionlink>__NEWSECTIONLINK__)|(?P<a_nonewsectionlink>__NONEWSECTIONLINK__)|(?P<a_hiddencat>__HIDDENCAT__)|(?P<a_expectunusedcategory>__EXPECTUNUSEDCATEGORY__)|(?P<a_index>__INDEX__)|(?P<a_noindex>__NOINDEX__)|(?P<a_staticredirect>__STATICREDIRECT__)|(?P<a_noglobal>__NOGLOBAL__)|(?P<a_disambiguation>__DISAMBIG__))__|\<\!\-\-\(\?\>\[\\s\\S\]\*\?\-\-\>\))*)(<nowiki>\s+<\/nowiki>)([^\n]*(?:\n|$))/Dim';

In T231569#5503164, @ssastry wrote:

In T231569#5502268, @mobrovac wrote:

@ssastry were you able to confirm Parsoid/PHP works as advertised on parsoid-php-beta.wmflabs.org ?

I will need to figure out how to test since that domain isn't configured anywhere in config. So, I imagine I need to use some host headers / proxies or something to actually test. Do you know how I might be able to test this?

In T231569#5502268, @mobrovac wrote:

@ssastry were you able to confirm Parsoid/PHP works as advertised on parsoid-php-beta.wmflabs.org ?

So, some other config seems to be off.

I Imagine some config is still using 8000 for testing whether the JS service is up. I don't know what that is and what needs fixing, but other than that, looks like the code got deployed. I'll verify Parsoid/JS manually shortly.

22:00:42 Started deploy [parsoid/deploy@7d2519e] (beta)
22:00:42 Deploying Rev: HEAD = 7d2519e6cf3155819ca7ddb75c81104fb62c37fe
22:00:42 Started deploy [parsoid/deploy@7d2519e] (beta): Updating Parsoid to 6fbdd703 (T231569)
22:00:42 
== DEFAULT ==
:* deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
:* deployment-parsoid09.deployment-prep.eqiad.wmflabs
parsoid/deploy: fetch stage(s): 100% (ok: 2; fail: 0; left: 0)                  
parsoid/deploy: config_deploy stage(s): 100% (ok: 2; fail: 0; left: 0)          
22:03:03 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'parsoid/deploy', '-g', 'default', 'promote', '--refresh-config'] on deployment-parsoid09.deployment-prep.eqiad.wmflabs returned [70]: Linking config files at: /srv/deployment/parsoid/deploy-cache/revs/7d2519e6cf3155819ca7ddb75c81104fb62c37fe/.git/config-files
Check depool is empty and will not be run
Check repool is empty and will not be run
Restarting service 'parsoid'
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Unhandled error:
deploy-local failed: <OSError> {}

I am going to decline this since we are going to be in Parsoid/PHP land soon enough and this won't be relevant anymore.

This looks resolved based on the new test run on scandium. After about 5 minutes of the test run, I am already noticing that a number of previously crashing tests are completing quickly.

In T232222#5495884, @ssastry wrote:

I followed instructions on https://www.mediawiki.org/wiki/Excimer#Profile_a_process_and_produce_a_flame_graph and used Excimer to generate a flamegraph while running html2wt on en:Berlin. I added the instrumentation locally to bin/parse.php and ran php bin/parse.php --html2wt < /tmp/berlin.html > /tmp/berlin.wt
I am uploading the generated flame graph here.

berlin.html2wt.profile.svg288 KBDownload

Looking at the flamegraph, it looks like this is similar to T232180: Zest.php: Pagebundle routes timeout for some pages and perhaps https://gerrit.wikimedia.org/r/c/mediawiki/libs/RemexHtml/+/535291 might help with this.

In T232222#5497026, @ssastry wrote:

In T232222#5495884, @ssastry wrote:

... perhaps https://gerrit.wikimedia.org/r/c/mediawiki/libs/RemexHtml/+/535291 might help with this.

No, that won't help because (a) the DOM has id attributes set => the shortcut to avoid checking for id conflicts doesn't apply (b) id conflicts are extremely unlikely => getElementById(..) for the new id will return null, but since we don't trust that, we go and do a full document search anyway.
The better workaround for both the phab tasks is probably https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/535290#message-40b70cf4985a0151e661f557ead69a7f0dfa59da

In T232180#5496256, @ssastry wrote:

The solution in https://gerrit.wikimedia.org/r/535290 and followups to avoid invoking getElementById fails because cite introduces a number id attributes and so we fallback to the full search.

In T232222#5495884, @ssastry wrote:

... perhaps https://gerrit.wikimedia.org/r/c/mediawiki/libs/RemexHtml/+/535291 might help with this.

The solution in https://gerrit.wikimedia.org/r/535290 and followups to avoid invoking getElementById fails because cite introduces a number id attributes and so we fallback to the full search.

I followed instructions on https://www.mediawiki.org/wiki/Excimer#Profile_a_process_and_produce_a_flame_graph and used Excimer to generate a flamegraph while running html2wt on en:Berlin. I added the instrumentation locally to bin/parse.php and ran php bin/parse.php --html2wt < /tmp/berlin.html > /tmp/berlin.wt

In T232538#5492336, @ssastry wrote:

I think that is worth following up on a separate phab ticket.

Ok, I verified that the instance has both Parsoid and Mediawiki latest version of code on there. There are gerrit patches to make this instance a mediawiki and parsoid scap target, but those can proceed independently of this.

In T232538#5492320, @ssastry wrote:

Next update.
I applied all those roles and ran sudo puppet agent -tv on the instance and it failed soon enough:

ssastry@deployment-mediawiki-parsoid10:~$ sudo puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find template 'varnish/errorpage.body.html.erb' at /etc/puppet/modules/profile/manifests/mediawiki/hhvm.pp:97:20 on node deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

So, I am stalled here at this point and need to hand it off to someone who knows how to proceed.

Next update.

In T232538#5492275, @hashar wrote:

That puppet prefix is there to not have to remember to apply those classes! Their removal defeat the purpose of the puppet prefix entirely.
If role::mediawiki::appserver and role::beta::mediawiki breaks the instance provisioning, I guess we have to fix them? Have you captured any log?
@jeena do you remember facing issues with puppet when you have created your deployment-mediawiki instance?

Okay, a further update.

An update.

In T232538#5490768, @gerritbot wrote:

Change 536598 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/puppet@production] beta cluster: Make deployment-parsoid09 a Mediawiki appserver as well
https://gerrit.wikimedia.org/r/536598

Joe deployed the revert and the puppet patch and I confirmed that this now directs the logs to 'parsoid-tests' type in logstash!

In T227209#5485875, @sbassett wrote:

Update: @Reedy and I are having a look at this and T230140 and should have some reports soon. We're going to timebox to maybe two weeks or so - hopefully that doesn't push back too much on any targeted deployment dates.

Hmm ... better, but there is still something to investigate. If this is not zest.php / getElementById, we should close this and open a new ticket. But, I am running out, so reporting the results for now:

ssastry@scandium:/srv/deployment/parsoid/deploy/src$ time curl -x scandium.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/transform/wikitext/to/pagebundle/Ken%20Schrader/913657403 -X POST < /dev/null > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1431k    0 1431k    0     0  11228      0 --:--:--  0:02:10 --:--:--  340k

Looks like this code at the end of modules/profile/files/logstash/filter-mediawiki.conf in the puppet repo might do the trick if we revert my mediawiki-config change patch?

# Scandium is a Parsoid test server and treat errors here as
# parsoid-tests errors and don't clog the main mediawiki channel
if [host] == "scandium" {
  mutate {
    replace => [ "type", "parsoid-tests" ]
  }
}

We have a workaround for testing purposes. So resolving this. The bigger qn. that this triggered is covered in T232563.

So, it looks like these logs disappeared from the "mediawiki" type, but they did not make an appearance under the "parsoid-tests" type in logstash (I searched in Kibana). Based on IRC conversations on MediaWiki-General, I learnt that there is puppet code involved in directing the log events from mediawiki to logstash. So, it looks like we'll need some puppet updates to these filters to ensure these events end up in logstash / kibana.

In T217867#5484727, @jlinehan wrote:

In T217867#5484496, @ssastry wrote:

@jlinehan, Early heads up from a project mgmt. POV.

Thanks for the heads up. I have had my plate completely full with Better Use of Data work this last quarter, so I haven't had a chance to finish off evaluating this library. I can make some time in Q2 to make sure it gets there so it's easier for your team to make a decision about building on it.

In T217867#5484661, @Tgr wrote:

https://github.com/ivopetkov/html5-dom-document-php was mentioned in the last JetBrains newsletter as a similar library.
(Wouldn't it be nice if our libraries would be mentioned in major newsletters as well? waves at T171073 and ducks)

@jlinehan, Early heads up from a project mgmt. POV. Once we are done with the porting and deployment, we might want to more seriously evaluate this ported library as a replacement for the PHP DOM. Without evaluation, especially performance, I do not want to say we will definitely be using this, but given all the bandaids we've been applying on top of PHP DOM, it is more likely we will want to consider this. I leave that evaluation up to @cscott and you.

In T232538#5482002, @Joe wrote:

hi @ssastry just a clarification: how would we load the parsoid code, if it can't be merged in the wmf vendor repository? Same way we do on scandium?

Here is a possible workaround that seems to work:

ssastry@scandium:~$ curl -x scandium.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/G.729d/
<!doctype html><title>Redirect</title><a href="http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/G.729d/457575777">http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/G.729d/457575777</a>

git grep of mediawiki code shows that it comes from https://github.com/wikimedia/mediawiki/blob/master/includes/WebRequest.php#L1127-L1152

Will verify when we start a new test run.

https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/504795 might be relevant here.

In T231933#5478324, @mobrovac wrote:

In T231933#5470174, @ssastry wrote:

@Jhernandez @Mholloway Do the tests need to actually query Parsoid or can they hit RESTBase instead? If the latter, then, I don't think anything will be affected. Parsoid/JS wasn't exposed to the public internet either.

They hit RB, but it does matter because RB can only access the Parsoid instance local to it. This used to work because, as said above, Parsoid from beta was set up to also respond to requests for production wikis, but without that ability, RB will not be able to produce a meaningful response.

In T229074#5479601, @ssastry wrote:

In T229074#5479372, @JTannerWMF wrote:

Hey @ssastry when should we expect to start paying attention to this?

Once we deploy the code to beta cluster ( T231569 ), I'll ping you.

In T229074#5479372, @JTannerWMF wrote:

Hey @ssastry when should we expect to start paying attention to this?

We have to resolve what direction we want to go with this. This will get clearer as we start the parser integration process. But, for clarity, I've marked this stalled.

In T232183#5471363, @hashar wrote:

So I guess we need a configuration to have MediaWiki on scandium to NOT send logs to logstash since that is just for testing

In T232183#5471340, @hashar wrote:

That is definitely a Wikimedia-production-error log! Not sure why you removed the project tag and the stacktrace :]

In T232183#5471340, @hashar wrote:

That is definitely a Wikimedia-production-error log! Not sure why you removed the project tag and the stacktrace :]