Page MenuHomePhabricator

can't publish translation on cx2-testing and cx-ofb - Error converting HTML to wikitext: docserver-http: HTTP 0
Closed, ResolvedPublic

Description

can't publish translation on cx2-testing and cx-ofb - error Error converting HTML to wikitext: docserver-http: HTTP 0 appears after we click publish.




According to @santhosh this issue comes from the latest (ongoing) updates on parsoid.
I'm not sure how this is going to impact us or if we need to communicate this to the parsoid team, up to you to handle that @Pginer-WMF

Details

Due Date
Dec 5 2019, 6:30 PM
Related Gerrit Patches:
mediawiki/services/cxserver : master[DO NOT MERGE] Test for Parsoid/PHP

Event Timeline

Jpita created this task.Oct 28 2019, 10:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2019, 10:07 AM
Pginer-WMF triaged this task as Unbreak Now! priority.Oct 28 2019, 10:16 AM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptOct 28 2019, 10:16 AM

@ssastry bases on the above, it seems that latest changes in parsoid could break Content translation if deployed with the next train. Any thoughts on how to avoid that?

Pginer-WMF lowered the priority of this task from Unbreak Now! to Medium.Oct 28 2019, 11:58 AM

Based on input from @santhosh, it seems that the testing servers have been enabled for the new version of Parsoid to start testing the changes, but this won't affect production until it is explicitly activated.

Depending on timing, @Jpita may need to consider whether another test instance may be needed for his QA work since these test servers became unusabe as a result.

@Pginer-WMF do we know who enabled that configuration?

@mobrovac know anything about this?
@Pginer-WMF @Jpita is your test server hitting RESTBase in the beta cluster (presumably yes)?

@Jpita is your test server hitting RESTBase in the beta cluster (presumably yes)?

@KartikMistry some input on this?

HTTP 0 would suggest that there was a service crash of some sorts in the pipeline and as a result a malformed response was returned (hence the 0 status code). But, in order to investigate more, we'd need a reproduction case. Also, some questions that spring to mind:

  • Where is the cx-testing host located?
  • Which RESTBase / Parsoid instances is it talking to?
  • @Pginer-WMF says that the testing servers have been enabled for the new version of Parsoid to start testing the changes. How are you doing that?
  • @Pginer-WMF says that the testing servers have been enabled for the new version of Parsoid to start testing the changes. How are you doing that?

I don't think we did anything to explicitly enable those. My understanding from a conversation with Santhosh was that the testing instances were affected but production won't. So I guess the configuration is different in each case, but I don't know the details. @santhosh has been doing further investigation, and may be able to provide more details.

The issue is appearing when we publish content specifically when html from cx is converted to Wikitext. But curiously the error contains the successfully converted wikitext.

The exact place in the code where this error was thrown:

throw new \MWException( "docserver-http: HTTP {$response['code']}: {$response['body']}" );

this happens if restbase return code is not 200. So, restbase is doing its job correctly, except the return code is 0. Which, according to the documentation of VirualRestbaseClient in core is - code : HTTP response code or 0 if there was a serious cURL error

The restbase endpoint configured in cx2-testing is https://en.wikipedia.org/api/rest_v1

I am not able to reproduce the issue locally with same code and same restbase URL. So suspecting some enviroment issue in cx2-testing.

@santhosh is the cx testing server running PHP7? There were changes introduced to the curl layer in MW that require PHP7+ to run.

santhosh@language-cx2:~$ php --version
PHP 7.3.11-1~deb10u1 (cli) (built: Oct 26 2019 14:14:18) ( NTS )

It is 7.3.11. Is there any specific setup need to do here?

To produce the issue just visit https://cx2-testing.wmflabs.org/index.php/Main_Page?veaction=edit and VE will show error saying "Error loading data from server: apierror-visualeditor-docserver-http: Error contacting the Parsoid/RESTBase server (HTTP 0). Would you like to retry?"

It is 7.3.11. Is there any specific setup need to do here?

Perhaps the libcurl version is off then? What's the version on your host? For reference, in production (where VRS + libcurl works), the libcurl version is 7.52.1

To produce the issue just visit https://cx2-testing.wmflabs.org/index.php/Main_Page?veaction=edit and VE will show error saying "Error loading data from server: apierror-visualeditor-docserver-http: Error contacting the Parsoid/RESTBase server (HTTP 0). Would you like to retry?"

It does indeed produce the issue but the actual response returned to VE's API module is not visible on the client, so it's very hard for me to figure out what is going on exactly. You should look into your MW instance's logs, there should be clues here.

But, if, as you say, the response is good apart from the status code, that might suggest a problem with setting the response code in VRS/curl.

$  dpkg -l | grep curl
ii  curl                                 7.64.0-4                    amd64        command line tool for transferring data with URL syntax
ii  libcurl3-gnutls:amd64                7.64.0-4                    amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
ii  libcurl4:amd64                       7.64.0-4                    amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
ii  php-curl                             2:7.3+69                    all          CURL module for PHP [default]
ii  php7.3-curl                          7.3.11-1~deb10u1            amd64        CURL module for PHP
ii  python-pycurl                        7.43.0.2-0.1                amd64        Python bindings to libcurl
ii  python3-pycurl                       7.43.0.2-0.1                amd64        Python bindings to libcurl (Python 3)      

Ok so the constants we are using in code are present in libcurl, so that's not it. The next step would be to inspect the log entries of your MW instance.

santhosh@language-cx2:~$ php --version
PHP 7.3.11-1~deb10u1 (cli) (built: Oct 26 2019 14:14:18) ( NTS )

It is 7.3.11. Is there any specific setup need to do here?

Production is PHP 7.2. Do we know if core has been tested with PHP 7.3?

Production is PHP 7.2. Do we know if core has been tested with PHP 7.3?

Not as much as 7.2, but generally yes. See https://www.mediawiki.org/wiki/Compatibility and PHP 7.3 support.

KartikMistry renamed this task from can't publish translation on cx2-testing and cx-ofb - Error converting HTML to wikitext: docserver-http: HTTP 0 to Can't publish translation on cx2-testing and cx-ofb - Error converting HTML to wikitext: docserver-http: HTTP 0.Nov 11 2019, 8:09 AM
KartikMistry claimed this task.
KartikMistry renamed this task from Can't publish translation on cx2-testing and cx-ofb - Error converting HTML to wikitext: docserver-http: HTTP 0 to can't publish translation on cx2-testing and cx-ofb - Error converting HTML to wikitext: docserver-http: HTTP 0.Nov 11 2019, 8:26 AM
KartikMistry raised the priority of this task from Medium to High.

This root cause here might actually be T238161: [Regression pre-wmf.6] Cannot create a new page on Beta cluster shows error "Error contacting the Parsoid/RESTBase server (HTTP 500)" but in order to confirm it, we'd need someone to answer the above questions:

  • Where is the cx-testing host located?
  • Which RESTBase / Parsoid instances is it talking to?
  • @Pginer-WMF says that the testing servers have been enabled for the new version of Parsoid to start testing the changes. How are you doing that?

Additional question:

  • Are you hitting this error when creating new pages or only editing old ones? If the former, are you using transform/html/to/wikitext ?

This root cause here might actually be T238161: [Regression pre-wmf.6] Cannot create a new page on Beta cluster shows error "Error contacting the Parsoid/RESTBase server (HTTP 500)" but in order to confirm it, we'd need someone to answer the above questions:

  • Where is the cx-testing host located?

They are labs instances. cx2-testing.wmflabs.org and cx-ofb.wmflabs.org

  • Which RESTBase / Parsoid instances is it talking to?

Production.

Additional question:

  • Are you hitting this error when creating new pages or only editing old ones? If the former, are you using transform/html/to/wikitext ?

Both new and old.

I think yes, we are using transform/html/to/wikitext as per cxserver/lib/mw/MwApiRequest.js

Ok, so definitely the same thing as the aforementioned task plus possible other issues. Closing this one as a duplicate for now, and if things don't work after that is fixed, please open a new task.

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptNov 14 2019, 3:10 PM
KartikMistry reopened this task as Open.Nov 18 2019, 6:07 AM

Unknown unrecoverable error has occurred. Error details: Error converting HTML to wikitext: docserver-http: HTTP 0 still happening with Labs instances (cx-ofb, cx2-testing).

@KartikMistry could you please provide

  1. the exact steps to reproduce
  2. any relevant logs from the CX extension (i.e. those that get the 0 status code and send it on to the client)

@KartikMistry could you please provide

  1. the exact steps to reproduce
  2. any relevant logs from the CX extension (i.e. those that get the 0 status code and send it on to the client)

Yes, please. Without that info, we don't know what we are looking for. :) Also (or alternatively), do we have access to the CX server labs instances if we need to log in and poke around for logs and errors?

Jpita added a comment.Nov 18 2019, 4:13 PM
  1. the exact steps to reproduce
  1. go to https://cx2-testing.wmflabs.org/index.php
  2. login/create account
  3. go to https://cx2-testing.wmflabs.org/index.php/Special:ContentTranslation
  4. start a new translation from the suggestion tabs
  5. translate some sections, change some text to avoid getting an error of too much unmodified text
  6. click the cog next to the publish button and change to Personal Draft
  7. open the browser developer tools
  8. click the publish button
  9. see the error that shows on the page and the console

Thank you @Jpita . Could we also get the list of patches that was applied in order for both the service and the extension to use Parsoid/PHP?

Thank you @Jpita . Could we also get the list of patches that was applied in order for both the service and the extension to use Parsoid/PHP?

@santhosh Am I missing anything?

@mobrovac, I tried the steps that @Jpita provided, and am looking at the type:parsoid-php events in logstash and I see nothing. So, either Parsoid-PHP is converting the HTML to wikitext without errors and something else is going on after Parsoid's response OR Parsoid-PHP isn't even getting the request and is being dropped by RESTBase because of some configuration errors in CX.

This should not have any impact on this task. It is just a spec version update.

Correct

Change 551734 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/cxserver@master] [DO NOT MERGE] Test for Parsoid/PHP

https://gerrit.wikimedia.org/r/551734

There are numerous places in CXServer where you set the Accept header to an older version, please update all of them to 2.1.0

This is not sufficient - all routes need to have the header set, not just when you fetch the content. Please apply Gerrit 551734 instead and test it.

This is not sufficient - all routes need to have the header set, not just when you fetch the content. Please apply Gerrit 551734 instead and test it.

I've cherry-picked Gerrit 551734 to cxserver.wmflabs.org.

I've cherry-picked Gerrit 551734 to cxserver.wmflabs.org.

... and?

VE already works as expected with Parsoid-PHP, so, I don't have any reason to believe CX won't either. @Jpita @KartikMistry can one of you provide at least one of us (temporary) access to the server so we can get this config issues resolved and get to the actual testing and identify any CX-specific issues? That would be very helpful so we can move this along. Thanks!

I've cherry-picked Gerrit 551734 to cxserver.wmflabs.org.

I've updated PS2 of the patch, please apply it, restart the service and test.

I've updated PS2 of the patch, please apply it, restart the service and test.

Thanks! Testing it.

Also added @mobrovac to Language group, so you can login to language-cx2.eqiad.wmflabs / cx-ofb.eqiad.wmflabs and cxserver2.eqiad.wmflabs instances.

I've updated PS2 of the patch, please apply it, restart the service and test.

Thanks! Testing it.
Also added @mobrovac to Language group, so you can login to language-cx2.eqiad.wmflabs / cx-ofb.eqiad.wmflabs and cxserver2.eqiad.wmflabs instances.

Any update from the testing?

Change 551734 abandoned by Mobrovac:
[DO NOT MERGE] Test for Parsoid/PHP

https://gerrit.wikimedia.org/r/551734

KartikMistry set Due Date to Dec 5 2019, 6:30 PM.Dec 3 2019, 8:00 AM

I am going to unlink this from the parent task and resolve the parent task since we've already deployed CX everywhere and no one has really complained. :) You can figure out how to unbreak your testing meanwhile.

I'm able to publish articles on both cx2-testing and cx-ofb Labs instances.

Jpita closed this task as Resolved.Jan 15 2020, 10:51 PM
Jpita moved this task from Needs QA to Done on the Language-Team (Language-2020-January-March) board.