Page MenuHomePhabricator

Content Translation fails to load (and publish) articles at en.wikipedia.beta.wmflabs.org
Open, HighPublic

Description

No articles can be loaded for translation at http://en.wikipedia.beta.wmflabs.org . Probably a cxserver misonfiguration.

This was found by @Etonkovidova.

Also, publishing will fail due to old config for ContentTranslation restbase_url.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

This is how restbase is designed to load articles from Wiki in Beta. Need to check with restbase.

KartikMistry reopened this task as Open.Jul 14 2016, 7:09 AM

Reopening. This need to fix anyway.

I suggest to increase priority. Being able to test things on beta labs is important.

KartikMistry raised the priority of this task from Medium to High.Jul 14 2016, 7:36 AM

Change 298930 had a related patch set uploaded (by KartikMistry):
Beta: Fix restbase_url for ContentTranslation

https://gerrit.wikimedia.org/r/298930

KartikMistry renamed this task from Content Translation fails to load articles at en.wikipedia.beta.wmflabs.org to Content Translation fails to load (and publish) articles at en.wikipedia.beta.wmflabs.org.Jul 14 2016, 11:24 AM
KartikMistry updated the task description. (Show Details)

Change 298947 had a related patch set uploaded (by KartikMistry):
Beta: Fix cxserver restbase_url

https://gerrit.wikimedia.org/r/298947

Change 298930 merged by jenkins-bot:
Beta: Remove restbase_url for ContentTranslation

https://gerrit.wikimedia.org/r/298930

This is blocked by T138088

Amire80 moved this task from Backlog to Failures on the ContentTranslation-Deployments board.
greg added a comment.Jul 28 2016, 4:34 PM

This is blocked by T138088

then it should be a subtask, not a parent task, right? :)

Change 298947 merged by Alexandros Kosiaris:
Beta: Fix cxserver restbase_url

https://gerrit.wikimedia.org/r/298947

Change 306674 had a related patch set uploaded (by KartikMistry):
Beta: Fix cxserver restbase_url

https://gerrit.wikimedia.org/r/306674

Waiting for Puppet not updating properly on sca01, https://phabricator.wikimedia.org/T143065

Can we have a written status update here? It's been two weeks since the last one.

Change 306674 merged by Alexandros Kosiaris:
Remove cxserver restbase_url

https://gerrit.wikimedia.org/r/306674

I think I've run out of idea about this. Same config works fine with Labs, while not in Beta.

Nikerabbit added a comment.EditedSep 28 2016, 4:08 PM

The reason article loading does not work is that we are using the deployment-prep restbase, which expects domains like en.wikipedia.beta.wmflabs.org not en.wikipedia.org. If we want to use deployment-prep restbase, then we need to update CX extension configuration I believe. This would mean that articles with dependencies need to be copied to the beta wikis.

Or if we want to use the production restbase (which I believe we do, to restore previous behavior until it got broken), we need to override the url that by default comes from here. Since that line hardcodes the path, we have the same problem we had with CX extension. We cannot just set cxserver::restbase_uri here. Perhaps if we updated cxserver.pp and config.yaml.erb we could conditionally set on labs the default host, which hopefully would not be overwritten by default service config. It uses merge_config but the documentation does not say which one takes precedence.

It uses merge_config but the documentation does not say which one takes precedence.

The second parameter. See https://docs.ruby-lang.org/en/2.0.0/Hash.html#method-i-update

The reason article loading does not work is that we are using the deployment-prep restbase, which expects domains like en.wikipedia.beta.wmflabs.org not en.wikipedia.org. If we want to use deployment-prep restbase, then we need to update CX extension configuration I believe. This would mean that articles with dependencies need to be copied to the beta wikis.

That's the old "Should Beta be using only Beta resources or should it be allowed to use production ones". The rule favors the former although I think exceptions exist (but don't quote me on that).

Or if we want to use the production restbase (which I believe we do, to restore previous behavior until it got broken), we need to override the url that by default comes from here.

Overriding it, is not really possible. One can only override on a per host level in hiera, not per service. And there are other services on sca0X hosts that probably use the exact same config. It was done in https://gerrit.wikimedia.org/r/#/c/291857/ as a way to providing a unified way for all services to reach the 2 APIs. Having one service on a host reaching a different API endpoint than all the other services isn't really possible.

Since that line hardcodes the path, we have the same problem we had with CX extension. We cannot just set cxserver::restbase_uri here. Perhaps if we updated cxserver.pp and config.yaml.erb we could conditionally set on labs the default host, which hopefully would not be overwritten by default service config. It uses merge_config but the documentation does not say which one takes precedence.

Note that the merge_config function does a shallow merge and not a deep merge so the entire services stanza of https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb;aa9175dd99d693ff65f75b5fe0968875a98a2864$51 will need to be passed. Which kind of defeats the point of having a central point to configure that.

That being said, maybe it makes sense to move the 2 API stanzas to a new top level stanza (like the metrics one right above the services one), with a migration period obviously. Not sure what changes to service::runner that entails. @mobrovac what do you think ?

The alternative of using full_config => true as a parameter to service::node and passing a full configuration to the config parameter does exist, but it is in reality just a superset of the above and is even more brittle and requires more work, just to override 1 variable.

As a conclusion, my take would be first to estimate if it is possible to use Beta wikis in Beta. I have no estimation on the amount of work required to start using Beta wikis instead of production wikis however, should I assume it's out of the question ? In that case, specify the entirety of the services stanza in cxserver's config.yaml.erb, forcing production always for now and work with services to make it a bit better so that only the restbase_req method needs to be specified.

Beta vs Prod seems to be a recurrent topic / issue / stepping stone for CXServer. I have voiced my wishes and concerns elsewhere and multiple times, but have always been met with the "beta is not prod, and we need prod" argument, which, IMHO is just a way of saying that the team doesn't want to keep Beta up to date with the pages / templates / etc they need in order to test CXServer properly. Sure, it is easier to use prod directly, but then we have to ask ourselves sincerely - what is the point of having Beta in the first place? Using CXServer from Beta with data from prod is equivalent to using it from a local install or any other labs project for that matter, i.e. Beta brings no added value to the table if you don't use its data.

More concretely to the point of this ticket, as of T144542: Enable config deploys for service::node services it is now possible to define the service's configuration in the deploy repo, and I think we should move CXServer to it. That would allow deployers to set the values differently for prod and beta and to do so independently from other services. I will create a ticket for this and we can take it from there. Sounds reasonable?

Beta vs Prod seems to be a recurrent topic / issue / stepping stone for CXServer. I have voiced my wishes and concerns elsewhere and multiple times, but have always been met with the "beta is not prod, and we need prod" argument, which, IMHO is just a way of saying that the team doesn't want to keep Beta up to date with the pages / templates / etc they need in order to test CXServer properly. Sure, it is easier to use prod directly, but then we have to ask ourselves sincerely - what is the point of having Beta in the first place? Using CXServer from Beta with data from prod is equivalent to using it from a local install or any other labs project for that matter, i.e. Beta brings no added value to the table if you don't use its data.

Agreed on the principle, but I can understand that keeping Beta up to date with the required pages / templates /etc maybe too much of a burden, time consuming etc. Well at least the first time, I suppose it can be automated.

More concretely to the point of this ticket, as of T144542: Enable config deploys for service::node services it is now possible to define the service's configuration in the deploy repo, and I think we should move CXServer to it. That would allow deployers to set the values differently for prod and beta and to do so independently from other services. I will create a ticket for this and we can take it from there. Sounds reasonable?

Fine by me

https://phabricator.wikimedia.org/T147634 should be first step to go forward in this direction. I'll check with @mobrovac about it.

Restricted Application edited projects, added ContentTranslation; removed CX-deployments. · View Herald TranscriptJun 13 2018, 6:34 PM
Aklapper removed KartikMistry as the assignee of this task.Jun 19 2020, 4:18 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)