| • Jpita | |
| Oct 28 2019, 10:07 AM |
| F30907106: image.png | |
| Oct 28 2019, 10:07 AM |
| F30907103: image.png | |
| Oct 28 2019, 10:07 AM |
| F30907108: image.png | |
| Oct 28 2019, 10:07 AM |
can't publish translation on cx2-testing and cx-ofb - error Error converting HTML to wikitext: docserver-http: HTTP 0 appears after we click publish.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| [DO NOT MERGE] Test for Parsoid/PHP | mediawiki/services/cxserver | master | +5 -12 |
@ssastry bases on the above, it seems that latest changes in parsoid could break Content translation if deployed with the next train. Any thoughts on how to avoid that?
Based on input from @santhosh, it seems that the testing servers have been enabled for the new version of Parsoid to start testing the changes, but this won't affect production until it is explicitly activated.
Depending on timing, @Jpita may need to consider whether another test instance may be needed for his QA work since these test servers became unusabe as a result.
@mobrovac know anything about this?
@Pginer-WMF @Jpita is your test server hitting RESTBase in the beta cluster (presumably yes)?
HTTP 0 would suggest that there was a service crash of some sorts in the pipeline and as a result a malformed response was returned (hence the 0 status code). But, in order to investigate more, we'd need a reproduction case. Also, some questions that spring to mind:
I don't think we did anything to explicitly enable those. My understanding from a conversation with Santhosh was that the testing instances were affected but production won't. So I guess the configuration is different in each case, but I don't know the details. @santhosh has been doing further investigation, and may be able to provide more details.
The issue is appearing when we publish content specifically when html from cx is converted to Wikitext. But curiously the error contains the successfully converted wikitext.
The exact place in the code where this error was thrown:
throw new \MWException( "docserver-http: HTTP {$response['code']}: {$response['body']}" );
this happens if restbase return code is not 200. So, restbase is doing its job correctly, except the return code is 0. Which, according to the documentation of VirualRestbaseClient in core is - code : HTTP response code or 0 if there was a serious cURL error
The restbase endpoint configured in cx2-testing is https://en.wikipedia.org/api/rest_v1
I am not able to reproduce the issue locally with same code and same restbase URL. So suspecting some enviroment issue in cx2-testing.
@santhosh is the cx testing server running PHP7? There were changes introduced to the curl layer in MW that require PHP7+ to run.
santhosh@language-cx2:~$ php --version PHP 7.3.11-1~deb10u1 (cli) (built: Oct 26 2019 14:14:18) ( NTS )
It is 7.3.11. Is there any specific setup need to do here?
To produce the issue just visit https://cx2-testing.wmflabs.org/index.php/Main_Page?veaction=edit and VE will show error saying "Error loading data from server: apierror-visualeditor-docserver-http: Error contacting the Parsoid/RESTBase server (HTTP 0). Would you like to retry?"
Perhaps the libcurl version is off then? What's the version on your host? For reference, in production (where VRS + libcurl works), the libcurl version is 7.52.1
To produce the issue just visit https://cx2-testing.wmflabs.org/index.php/Main_Page?veaction=edit and VE will show error saying "Error loading data from server: apierror-visualeditor-docserver-http: Error contacting the Parsoid/RESTBase server (HTTP 0). Would you like to retry?"
It does indeed produce the issue but the actual response returned to VE's API module is not visible on the client, so it's very hard for me to figure out what is going on exactly. You should look into your MW instance's logs, there should be clues here.
But, if, as you say, the response is good apart from the status code, that might suggest a problem with setting the response code in VRS/curl.
$ dpkg -l | grep curl ii curl 7.64.0-4 amd64 command line tool for transferring data with URL syntax ii libcurl3-gnutls:amd64 7.64.0-4 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour) ii libcurl4:amd64 7.64.0-4 amd64 easy-to-use client-side URL transfer library (OpenSSL flavour) ii php-curl 2:7.3+69 all CURL module for PHP [default] ii php7.3-curl 7.3.11-1~deb10u1 amd64 CURL module for PHP ii python-pycurl 7.43.0.2-0.1 amd64 Python bindings to libcurl ii python3-pycurl 7.43.0.2-0.1 amd64 Python bindings to libcurl (Python 3)
Ok so the constants we are using in code are present in libcurl, so that's not it. The next step would be to inspect the log entries of your MW instance.
Production is PHP 7.2. Do we know if core has been tested with PHP 7.3?
Not as much as 7.2, but generally yes. See https://www.mediawiki.org/wiki/Compatibility and PHP 7.3 support.
This root cause here might actually be T238161: [Regression pre-wmf.6] Cannot create a new page on Beta cluster shows error "Error contacting the Parsoid/RESTBase server (HTTP 500)" but in order to confirm it, we'd need someone to answer the above questions:
Additional question:
They are labs instances. cx2-testing.wmflabs.org and cx-ofb.wmflabs.org
- Which RESTBase / Parsoid instances is it talking to?
Production.
Additional question:
- Are you hitting this error when creating new pages or only editing old ones? If the former, are you using transform/html/to/wikitext ?
Both new and old.
I think yes, we are using transform/html/to/wikitext as per cxserver/lib/mw/MwApiRequest.js
Ok, so definitely the same thing as the aforementioned task plus possible other issues. Closing this one as a duplicate for now, and if things don't work after that is fixed, please open a new task.
Unknown unrecoverable error has occurred. Error details: Error converting HTML to wikitext: docserver-http: HTTP 0 still happening with Labs instances (cx-ofb, cx2-testing).
@KartikMistry could you please provide
Yes, please. Without that info, we don't know what we are looking for. :) Also (or alternatively), do we have access to the CX server labs instances if we need to log in and poke around for logs and errors?
Thank you @Jpita . Could we also get the list of patches that was applied in order for both the service and the extension to use Parsoid/PHP?
@santhosh Am I missing anything?
@mobrovac, I tried the steps that @Jpita provided, and am looking at the type:parsoid-php events in logstash and I see nothing. So, either Parsoid-PHP is converting the HTML to wikitext without errors and something else is going on after Parsoid's response OR Parsoid-PHP isn't even getting the request and is being dropped by RESTBase because of some configuration errors in CX.
This should not have any impact on this task. It is just a spec version update.
- cxserver is cherry-picked with: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/cxserver/+/546908/
Correct
Change 551734 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/cxserver@master] [DO NOT MERGE] Test for Parsoid/PHP
There are numerous places in CXServer where you set the Accept header to an older version, please update all of them to 2.1.0
- cxserver is cherry-picked with: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/cxserver/+/546908/
This is not sufficient - all routes need to have the header set, not just when you fetch the content. Please apply Gerrit 551734 instead and test it.
VE already works as expected with Parsoid-PHP, so, I don't have any reason to believe CX won't either. @Jpita @KartikMistry can one of you provide at least one of us (temporary) access to the server so we can get this config issues resolved and get to the actual testing and identify any CX-specific issues? That would be very helpful so we can move this along. Thanks!
Thanks! Testing it.
Also added @mobrovac to Language group, so you can login to language-cx2.eqiad.wmflabs / cx-ofb.eqiad.wmflabs and cxserver2.eqiad.wmflabs instances.
I am going to unlink this from the parent task and resolve the parent task since we've already deployed CX everywhere and no one has really complained. :) You can figure out how to unbreak your testing meanwhile.