Page MenuHomePhabricator

Create a staging environment for OCG/Proton
Closed, ResolvedPublic8 Estimated Story Points

Description

There's currently no shared environment for the Product Owner (@ovasileva), Designer (@Nirzar), or engineers to test changes to Extension:Collection prior to merging them or sign off on changes before they're deployed.

AC

  • There exists a staging server specifically for Reading Web/Infra to stage, test, and sign off on changes.
  • The staging server is publicly accessible at http://proton-staging.wmflabs.org/
  • The Reading Web engineers and @Tgr have SSH access to the server.

Developer Notes

  1. Follow these sections of https://www.mediawiki.org/wiki/Reading/Web/Setting_up_a_staging_environment:
    • Set up the instance
      • Only vagrant roles enable offline interwiki. No other roles are necessary.
    • Set up regular codebase updates
  2. The most up-to-date guide on setting up a development instance of OCG is here: https://wikitech.wikimedia.org/wiki/OCG#Installing_a_development_instance.

Questions

@phuedx: No. I don't want to break staging or over-complicate its current function by adding more roles when it should be fairly cheap to spin up a new server.

  • Should we configure OCG, Extension:Collection and the PDF renderer to fetch page content via the production RESTBase instance?

This hasn't been addressed. However, it's possible to do with a minor config change: T174430#3589455

Event Timeline

Added interwiki role as we would require that to rest RTL languages

phuedx updated the task description. (Show Details)
phuedx updated the task description. (Show Details)
phuedx removed the point value for this task.
ovasileva set the point value for this task to 8.Aug 30 2017, 4:59 PM

Just a quick update below.

Besides the proposed roles, I've enabled the following ones too. Doing so fixed the presentation of the test article.

* cite
* parserfunctions
* scribunto

I've also imported an article and a couple dozen pages from Wikipedia: Book.

OCG book generation page is failing with some error though. I'll have to debug and fix that.

I have the password of the wiki's admin account. Ping me via email and I'll be happy to share. Or let me know your username and I'll give you some rights to do superuser stuff.

I have the password of the wiki's admin account. Ping me via email and I'll be happy to share. Or let me know your username and I'll give you some rights to do superuser stuff.

Or you can just SSH in and run createAndPromote.php :-)

Thanks for the tip, @Tgr. On another note, do you know what maybe causing T94308: Download as PDF: Rendering failed with non zero code on http://proton-staging.wmflabs.org/w/index.php?title=Special:Book&bookcmd=rendering&return_to=Special%3ABook&collection_id=8f18602d053200fb62561a48a89af8ad38e3119d&writer=rdf2latex ?

The logs have this entry:

{"name":"mw-ocg-service","hostname":"mediawiki-vagrant","pid":1162,"level":50,"channel":"backend.bundler.error","job":{"id":"8f18602d053200fb62561a48a89af8ad38e3119d","writer":"rdf2latex"},"err":{"message":"Bundling process died with non zero code: 1","name":"BundlerError","stack":"BundlerError: Bundling process died with non zero code: 1\n    at /vagrant/srv/mw-ocg-service/lib/threads/backend.js:563:11\n    at tryCatch1 (/vagrant/srv/mw-ocg-service/node_modules/babybird/lib/promise.js:40:12)\n    at promiseReactionJob (/vagrant/srv/mw-ocg-service/node_modules/babybird/lib/promise.js:269:19)\n    at PromiseReactionJobTask.call (/vagrant/srv/mw-ocg-service/node_modules/babybird/lib/promise.js:284:3)\n    at flush (/vagrant/srv/mw-ocg-service/node_modules/babybird/node_modules/asap/raw.js:50:29)\n    at _combinedTickCallback (internal/process/next_tick.js:73:7)\n    at process._tickCallback (internal/process/next_tick.js:104:9)"},"msg":"Bundling process died with non zero code: 1","time":"2017-09-05T19:34:12.163Z","v":0}

Any ideas how to fix it?

Krinkle renamed this task from Create a staging environment to Create a staging environment for OCG/Proton.Sep 5 2017, 8:06 PM

The more helpful log entry is

{"name":"mw-ocg-service","hostname":"mediawiki-vagrant","pid":1162,"level":50,"channel":"backend.bundler.bin","job":{"id":"8f18602d053200fb62561a48a89af8ad38e3119d","writer":"rdf2latex"},"msg":"Error: Error fetching parsoid3 result: http://localhost:8000/proton-staging.wmflabs.org/v3/page/html/Book/10 500\n    at /vagrant/srv/mw-ocg-bundler/lib/parsoid.js:239:20\n    at /vagrant/srv/mw-ocg-bundler/lib/retry-request.js:77:3\n    at /vagrant/srv/mw-ocg-bundler/node_modules/prfun/lib/index.js:596:20\n    at tryCatch1 (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/lib/promise.js:40:12)\n    at promiseReactionJob (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/lib/promise.js:269:19)\n    at PromiseReactionJobTask.call (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/lib/promise.js:284:3)\n    at flush (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/node_modules/asap/raw.js:50:29)\n    at _combinedTickCallback (internal/process/next_tick.js:73:7)\n    at process._tickCallback (internal/process/next_tick.js:104:9)","time":"2017-09-05T19:34:12.157Z","v":0}

and from the Parsoid log

{"name":"parsoid","hostname":"mediawiki-vagrant","pid":1192,"level":60,"err":{"message":"Config Request failure for \"http://localhost/w/api.php\": Error: connect ECONNREFUSED 127.0.0.1:80","name":"parsoid","stack":"parsoid: Config Request failure for \"http://localhost/w/api.php\": Error: connect ECONNREFUSED 127.0.0.1:80\n    at ConfigRequest.ApiRequest._requestCB (/vagrant/srv/parsoid/lib/mw/ApiRequest.js:415:15)\n    at self.callback (/vagrant/srv/parsoid/node_modules/request/request.js:188:22)\n    at emitOne (events.js:96:13)\n    at Request.emit (events.js:188:7)\n    at Request.onRequestError (/vagrant/srv/parsoid/node_modules/request/request.js:884:8)\n    at emitOne (events.js:101:20)\n    at ClientRequest.emit (events.js:188:7)\n    at Socket.socketErrorListener (_http_client.js:310:9)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at emitErrorNT (net.js:1277:8)\n    at _combinedTickCallback (internal/process/next_tick.js:80:11)\n    at process._tickCallback (internal/process/next_tick.js:104:9)","levelPath":"fatal/request"},"msg":"Config Request failure for \"http://localhost/w/api.php\": Error: connect ECONNREFUSED 127.0.0.1:80","time":"2017-09-05T20:29:40.044Z","v":0}
{"name":"parsoid","hostname":"mediawiki-vagrant","pid":1192,"level":40,"logType":"warn/api/econnrefused","wiki":"localhost","title":"Book","oldId":null,"reqId":null,"userAgent":"curl/7.38.0","msg":"Failed API request, {\"error\":{\"code\":\"ECONNREFUSED\",\"errno\":\"ECONNREFUSED\",\"syscall\":\"connect\",\"address\":\"127.0.0.1\",\"port\":80},\"retries-remaining\":1}","longMsg":"Failed API request,\n{\"error\":{\"code\":\"ECONNREFUSED\",\"errno\":\"ECONNREFUSED\",\"syscall\":\"connect\",\"address\":\"127.0.0.1\",\"port\":80},\"retries-remaining\":1}","levelPath":"warn/api/econnrefused","time":"2017-09-05T22:44:05.503Z","v":0}

The correct MW API URL is http://localhost:8080/w/api.php. I tried to fix this issue for installations with non-default ports with https://gerrit.wikimedia.org/r/#/c/375933/ , maybe that broke it somehow for the default port? /vagrant/srv/parsoid/config.vagrant.yaml has http://localhost:8080/w/api.php for the MW API URL though, I am not sure where the :80 request is coming from.

So apparently ::forwarded_port and ::port_fragment are the same on normal Vagrant but different in Labs, and that breaks a bunch of things, including RESTBase. Patched puppet locally, tracking the writing of a proper fix in T175100.

OCG works now (had to restart the host first due to node hanging up).

Is there anything in particular you need from developers with respect to "code review" or can we move this to sign off ?

Will git auto update here or do we have to do it manually?

bmansurov added a subscriber: bmansurov.

I haven't setup any auto update for the sake of stable and predictable behavior. We'll have to maintain the instance manually.

Prior to resolving this task, would it be worth documenting what the had to do to set up the staging environment on a wiki (next to Setting up a staging environment, perhaps?)

I'd rather we don't as that document will get outdated without active maintenance. Things are already well documentaed at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS.

ovasileva moved this task from Current Sprint to Resolved on the Proton board.

Will git auto update here or do we have to do it manually?

vagrant git-update will work the same way as on any other normal vagrant box (ie. it will do the update but will skip repos which are not on master or have uncommitted changes; for repos which are on master but have diverged, it will try a rebase and might need manual cleanup).


Note that Collection behaves somewhat erratically on Vagrant (or probably any kind of local installation): you get error messages when trying to add pages, but usually they get added anyway. I didn't want to spend time on tracking that down as it's going to be rewritten anyway, but in any case it's not specific to the staging environment.

Change 376487 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/vagrant@master] Add ElectronVirtualRestService config to offline role

https://gerrit.wikimedia.org/r/376487

I'd rather we don't as that document will get outdated without active maintenance. Things are already well documented at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS.

I agree that the general process of setting up labs instance is well documented there as well as a few gotchas – indeed, it's linked from the MobileFrontend-specific doc.

I'm not sure if there are gotchas specific to the offline role that might be worth writing down. If you think not, then fine.

I think we can try and link this ticket at most. Even the offline role is changing, so things may get old and useless later.

Looks like Sam might be signing this off? Please re-assign if that's not the case.

Most problems that came up should be fixed in Vagrant, not documented.

Maybe the one thing worth mentioning is that you can use production RESTBase (for comparing PDF output to production, for example) with a setting like this:

$wgVirtualRestConfig['modules']['restbase'] = [
        'url' => 'https://en.wikipedia.org/api/rest_v1/',
        'domain' => 'en.wikipedia.org',
        'forwardCookies' => false,
        'fixedUrl' => true,
] + $wgVirtualRestConfig['modules']['restbase'];

(The local MW API will be used for sections etc. so you'll need to have the same wikitext locally, but the HTML will be fetched from enwiki.)

phuedx updated the task description. (Show Details)
phuedx updated the task description. (Show Details)

As well as verifying that I could download a page as a one or two column PDF, I was bold and created mw:Reading/Web/Setting_up_a_Proton_staging_environment. It's simple and isn't tightly coupled to any MWV issue that y'all have encountered while working on this task.

Change 376487 merged by jenkins-bot:
[mediawiki/vagrant@master] Add ElectronVirtualRestService config to offline role

https://gerrit.wikimedia.org/r/376487