Page MenuHomePhabricator

Content Translation fails to load (and publish) articles at en.wikipedia.beta.wmflabs.org
Open, HighPublic

Description

No articles can be loaded for translation at http://en.wikipedia.beta.wmflabs.org . Probably a cxserver misonfiguration.

This was found by @Etonkovidova.

Also, publishing will fail due to old config for ContentTranslation restbase_url.

Details

Related Gerrit Patches:
operations/puppet : productionRemove cxserver restbase_url
operations/puppet : productionBeta: Fix cxserver restbase_url
operations/mediawiki-config : masterBeta: Remove restbase_url for ContentTranslation

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Amire80 moved this task from CX8 to Bugs on the ContentTranslation board.Apr 20 2016, 1:14 PM
Amire80 triaged this task as Normal priority.Apr 20 2016, 1:30 PM
KartikMistry closed this task as Invalid.Apr 22 2016, 9:57 AM

This is how restbase is designed to load articles from Wiki in Beta. Need to check with restbase.

KartikMistry reopened this task as Open.Jul 14 2016, 7:09 AM

Reopening. This need to fix anyway.

I suggest to increase priority. Being able to test things on beta labs is important.

KartikMistry raised the priority of this task from Normal to High.Jul 14 2016, 7:36 AM

Change 298930 had a related patch set uploaded (by KartikMistry):
Beta: Fix restbase_url for ContentTranslation

https://gerrit.wikimedia.org/r/298930

KartikMistry renamed this task from Content Translation fails to load articles at en.wikipedia.beta.wmflabs.org to Content Translation fails to load (and publish) articles at en.wikipedia.beta.wmflabs.org.Jul 14 2016, 11:24 AM
KartikMistry updated the task description. (Show Details)

Change 298947 had a related patch set uploaded (by KartikMistry):
Beta: Fix cxserver restbase_url

https://gerrit.wikimedia.org/r/298947

Change 298930 merged by jenkins-bot:
Beta: Remove restbase_url for ContentTranslation

https://gerrit.wikimedia.org/r/298930

This is blocked by T138088

Amire80 moved this task from Backlog to Failures on the ContentTranslation-Deployments board.
greg added a comment.Jul 28 2016, 4:34 PM

This is blocked by T138088

then it should be a subtask, not a parent task, right? :)

Change 298947 merged by Alexandros Kosiaris:
Beta: Fix cxserver restbase_url

https://gerrit.wikimedia.org/r/298947

Change 306674 had a related patch set uploaded (by KartikMistry):
Beta: Fix cxserver restbase_url

https://gerrit.wikimedia.org/r/306674

Waiting for Puppet not updating properly on sca01, https://phabricator.wikimedia.org/T143065

Can we have a written status update here? It's been two weeks since the last one.

Change 306674 merged by Alexandros Kosiaris:
Remove cxserver restbase_url

https://gerrit.wikimedia.org/r/306674

I think I've run out of idea about this. Same config works fine with Labs, while not in Beta.

Nikerabbit added a comment.EditedSep 28 2016, 4:08 PM

The reason article loading does not work is that we are using the deployment-prep restbase, which expects domains like en.wikipedia.beta.wmflabs.org not en.wikipedia.org. If we want to use deployment-prep restbase, then we need to update CX extension configuration I believe. This would mean that articles with dependencies need to be copied to the beta wikis.

Or if we want to use the production restbase (which I believe we do, to restore previous behavior until it got broken), we need to override the url that by default comes from here. Since that line hardcodes the path, we have the same problem we had with CX extension. We cannot just set cxserver::restbase_uri here. Perhaps if we updated cxserver.pp and config.yaml.erb we could conditionally set on labs the default host, which hopefully would not be overwritten by default service config. It uses merge_config but the documentation does not say which one takes precedence.

It uses merge_config but the documentation does not say which one takes precedence.

The second parameter. See https://docs.ruby-lang.org/en/2.0.0/Hash.html#method-i-update

The reason article loading does not work is that we are using the deployment-prep restbase, which expects domains like en.wikipedia.beta.wmflabs.org not en.wikipedia.org. If we want to use deployment-prep restbase, then we need to update CX extension configuration I believe. This would mean that articles with dependencies need to be copied to the beta wikis.

That's the old "Should Beta be using only Beta resources or should it be allowed to use production ones". The rule favors the former although I think exceptions exist (but don't quote me on that).

Or if we want to use the production restbase (which I believe we do, to restore previous behavior until it got broken), we need to override the url that by default comes from here.

Overriding it, is not really possible. One can only override on a per host level in hiera, not per service. And there are other services on sca0X hosts that probably use the exact same config. It was done in https://gerrit.wikimedia.org/r/#/c/291857/ as a way to providing a unified way for all services to reach the 2 APIs. Having one service on a host reaching a different API endpoint than all the other services isn't really possible.

Since that line hardcodes the path, we have the same problem we had with CX extension. We cannot just set cxserver::restbase_uri here. Perhaps if we updated cxserver.pp and config.yaml.erb we could conditionally set on labs the default host, which hopefully would not be overwritten by default service config. It uses merge_config but the documentation does not say which one takes precedence.

Note that the merge_config function does a shallow merge and not a deep merge so the entire services stanza of https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/service/templates/node/config.yaml.erb;aa9175dd99d693ff65f75b5fe0968875a98a2864$51 will need to be passed. Which kind of defeats the point of having a central point to configure that.

That being said, maybe it makes sense to move the 2 API stanzas to a new top level stanza (like the metrics one right above the services one), with a migration period obviously. Not sure what changes to service::runner that entails. @mobrovac what do you think ?

The alternative of using full_config => true as a parameter to service::node and passing a full configuration to the config parameter does exist, but it is in reality just a superset of the above and is even more brittle and requires more work, just to override 1 variable.

As a conclusion, my take would be first to estimate if it is possible to use Beta wikis in Beta. I have no estimation on the amount of work required to start using Beta wikis instead of production wikis however, should I assume it's out of the question ? In that case, specify the entirety of the services stanza in cxserver's config.yaml.erb, forcing production always for now and work with services to make it a bit better so that only the restbase_req method needs to be specified.

Beta vs Prod seems to be a recurrent topic / issue / stepping stone for CXServer. I have voiced my wishes and concerns elsewhere and multiple times, but have always been met with the "beta is not prod, and we need prod" argument, which, IMHO is just a way of saying that the team doesn't want to keep Beta up to date with the pages / templates / etc they need in order to test CXServer properly. Sure, it is easier to use prod directly, but then we have to ask ourselves sincerely - what is the point of having Beta in the first place? Using CXServer from Beta with data from prod is equivalent to using it from a local install or any other labs project for that matter, i.e. Beta brings no added value to the table if you don't use its data.

More concretely to the point of this ticket, as of T144542: Enable config deploys for service::node services it is now possible to define the service's configuration in the deploy repo, and I think we should move CXServer to it. That would allow deployers to set the values differently for prod and beta and to do so independently from other services. I will create a ticket for this and we can take it from there. Sounds reasonable?

Beta vs Prod seems to be a recurrent topic / issue / stepping stone for CXServer. I have voiced my wishes and concerns elsewhere and multiple times, but have always been met with the "beta is not prod, and we need prod" argument, which, IMHO is just a way of saying that the team doesn't want to keep Beta up to date with the pages / templates / etc they need in order to test CXServer properly. Sure, it is easier to use prod directly, but then we have to ask ourselves sincerely - what is the point of having Beta in the first place? Using CXServer from Beta with data from prod is equivalent to using it from a local install or any other labs project for that matter, i.e. Beta brings no added value to the table if you don't use its data.

Agreed on the principle, but I can understand that keeping Beta up to date with the required pages / templates /etc maybe too much of a burden, time consuming etc. Well at least the first time, I suppose it can be automated.

More concretely to the point of this ticket, as of T144542: Enable config deploys for service::node services it is now possible to define the service's configuration in the deploy repo, and I think we should move CXServer to it. That would allow deployers to set the values differently for prod and beta and to do so independently from other services. I will create a ticket for this and we can take it from there. Sounds reasonable?

Fine by me

https://phabricator.wikimedia.org/T147634 should be first step to go forward in this direction. I'll check with @mobrovac about it.

Restricted Application edited projects, added ContentTranslation; removed CX-deployments. · View Herald TranscriptJun 13 2018, 6:34 PM