Create a staging environment for OCG/Proton
Closed, ResolvedPublic8 Estimated Story Points
Actions

Assigned To

Authored By

	phuedx
	Aug 29 2017, 11:56 AM

Description

There's currently no shared environment for the Product Owner (@ovasileva), Designer (@Nirzar), or engineers to test changes to Extension:Collection prior to merging them or sign off on changes before they're deployed.

AC

There exists a staging server specifically for Reading Web/Infra to stage, test, and sign off on changes.
The staging server is publicly accessible at http://proton-staging.wmflabs.org/
The Reading Web engineers and @Tgr have SSH access to the server.

Developer Notes

Follow these sections of https://www.mediawiki.org/wiki/Reading/Web/Setting_up_a_staging_environment:
- Set up the instance
  - Only vagrant roles enable offline interwiki. No other roles are necessary.
- Set up regular codebase updates
The most up-to-date guide on setting up a development instance of OCG is here: https://wikitech.wikimedia.org/wiki/OCG#Installing_a_development_instance.

Questions

Should we do this on the staging server?

@phuedx: No. I don't want to break staging or over-complicate its current function by adding more roles when it should be fairly cheap to spin up a new server.

Should we configure OCG, Extension:Collection and the PDF renderer to fetch page content via the production RESTBase instance?

This hasn't been addressed. However, it's possible to do with a minor config change: T174430#3589455

Details

	Subject	Repo	Branch	Lines +/-
	Add ElectronVirtualRestService config to offline role	mediawiki/vagrant	master	+26 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• JKatzWMF	T150871 [EPIC] (Proposal) Replicate core OCG features and sunset OCG service
		Resolved		phuedx	T174430 Create a staging environment for OCG/Proton

Event Timeline

phuedx created this task.Aug 29 2017, 11:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 29 2017, 11:56 AM

ovasileva triaged this task as High priority.Aug 29 2017, 12:02 PM

phuedx updated the task description. (Show Details)Aug 29 2017, 12:52 PM

phuedx edited projects, added Web-Team-Backlog; removed Web-Team-Backlog (Tracking).Aug 29 2017, 3:44 PM

ovasileva updated the task description. (Show Details)Aug 29 2017, 4:41 PM

• Niedzielski updated the task description. (Show Details)Aug 29 2017, 4:44 PM

pmiazga updated the task description. (Show Details)Aug 29 2017, 4:44 PM

ovasileva moved this task from Incoming to Needs Prioritization on the Web-Team-Backlog board.Aug 29 2017, 4:45 PM

Added interwiki role as we would require that to rest RTL languages

phuedx updated the task description. (Show Details)Aug 29 2017, 4:45 PM

phuedx updated the task description. (Show Details)

phuedx updated the task description. (Show Details)Aug 30 2017, 4:09 PM

ovasileva added a project: Readers-Web-Kanbanana-Board-Old.Aug 30 2017, 4:15 PM

ovasileva moved this task from Needs Prioritization to Upcoming on the Web-Team-Backlog board.

ovasileva set the point value for this task to 8.

phuedx updated the task description. (Show Details)Aug 30 2017, 4:16 PM

phuedx removed the point value for this task.

ovasileva added a project: Proton.Aug 30 2017, 4:53 PM

ovasileva moved this task from Triage to Backlog on the Proton board.Aug 30 2017, 4:56 PM

ovasileva set the point value for this task to 8.Aug 30 2017, 4:59 PM

Jdlrobson moved this task from Upcoming to 2017-18 Q1 on the Web-Team-Backlog board.Aug 31 2017, 1:40 PM

• bmansurov claimed this task.Sep 5 2017, 3:07 PM

• bmansurov moved this task from To Do to Doing on the Readers-Web-Kanbanana-Board-Old board.

Just a quick update below.

Besides the proposed roles, I've enabled the following ones too. Doing so fixed the presentation of the test article.

* cite
* parserfunctions
* scribunto

I've also imported an article and a couple dozen pages from Wikipedia: Book.

OCG book generation page is failing with some error though. I'll have to debug and fix that.

I have the password of the wiki's admin account. Ping me via email and I'll be happy to share. Or let me know your username and I'll give you some rights to do superuser stuff.

In T174430#3581494, @bmansurov wrote:

I have the password of the wiki's admin account. Ping me via email and I'll be happy to share. Or let me know your username and I'll give you some rights to do superuser stuff.

Or you can just SSH in and run createAndPromote.php :-)

Thanks for the tip, @Tgr. On another note, do you know what maybe causing T94308: Download as PDF: Rendering failed with non zero code on http://proton-staging.wmflabs.org/w/index.php?title=Special:Book&bookcmd=rendering&return_to=Special%3ABook&collection_id=8f18602d053200fb62561a48a89af8ad38e3119d&writer=rdf2latex ?

The logs have this entry:

{"name":"mw-ocg-service","hostname":"mediawiki-vagrant","pid":1162,"level":50,"channel":"backend.bundler.error","job":{"id":"8f18602d053200fb62561a48a89af8ad38e3119d","writer":"rdf2latex"},"err":{"message":"Bundling process died with non zero code: 1","name":"BundlerError","stack":"BundlerError: Bundling process died with non zero code: 1\n    at /vagrant/srv/mw-ocg-service/lib/threads/backend.js:563:11\n    at tryCatch1 (/vagrant/srv/mw-ocg-service/node_modules/babybird/lib/promise.js:40:12)\n    at promiseReactionJob (/vagrant/srv/mw-ocg-service/node_modules/babybird/lib/promise.js:269:19)\n    at PromiseReactionJobTask.call (/vagrant/srv/mw-ocg-service/node_modules/babybird/lib/promise.js:284:3)\n    at flush (/vagrant/srv/mw-ocg-service/node_modules/babybird/node_modules/asap/raw.js:50:29)\n    at _combinedTickCallback (internal/process/next_tick.js:73:7)\n    at process._tickCallback (internal/process/next_tick.js:104:9)"},"msg":"Bundling process died with non zero code: 1","time":"2017-09-05T19:34:12.163Z","v":0}

Any ideas how to fix it?

Krinkle renamed this task from Create a staging environment to Create a staging environment for OCG/Proton.Sep 5 2017, 8:06 PM

The more helpful log entry is

{"name":"mw-ocg-service","hostname":"mediawiki-vagrant","pid":1162,"level":50,"channel":"backend.bundler.bin","job":{"id":"8f18602d053200fb62561a48a89af8ad38e3119d","writer":"rdf2latex"},"msg":"Error: Error fetching parsoid3 result: http://localhost:8000/proton-staging.wmflabs.org/v3/page/html/Book/10 500\n    at /vagrant/srv/mw-ocg-bundler/lib/parsoid.js:239:20\n    at /vagrant/srv/mw-ocg-bundler/lib/retry-request.js:77:3\n    at /vagrant/srv/mw-ocg-bundler/node_modules/prfun/lib/index.js:596:20\n    at tryCatch1 (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/lib/promise.js:40:12)\n    at promiseReactionJob (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/lib/promise.js:269:19)\n    at PromiseReactionJobTask.call (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/lib/promise.js:284:3)\n    at flush (/vagrant/srv/mw-ocg-bundler/node_modules/babybird/node_modules/asap/raw.js:50:29)\n    at _combinedTickCallback (internal/process/next_tick.js:73:7)\n    at process._tickCallback (internal/process/next_tick.js:104:9)","time":"2017-09-05T19:34:12.157Z","v":0}

and from the Parsoid log

{"name":"parsoid","hostname":"mediawiki-vagrant","pid":1192,"level":60,"err":{"message":"Config Request failure for \"http://localhost/w/api.php\": Error: connect ECONNREFUSED 127.0.0.1:80","name":"parsoid","stack":"parsoid: Config Request failure for \"http://localhost/w/api.php\": Error: connect ECONNREFUSED 127.0.0.1:80\n    at ConfigRequest.ApiRequest._requestCB (/vagrant/srv/parsoid/lib/mw/ApiRequest.js:415:15)\n    at self.callback (/vagrant/srv/parsoid/node_modules/request/request.js:188:22)\n    at emitOne (events.js:96:13)\n    at Request.emit (events.js:188:7)\n    at Request.onRequestError (/vagrant/srv/parsoid/node_modules/request/request.js:884:8)\n    at emitOne (events.js:101:20)\n    at ClientRequest.emit (events.js:188:7)\n    at Socket.socketErrorListener (_http_client.js:310:9)\n    at emitOne (events.js:96:13)\n    at Socket.emit (events.js:188:7)\n    at emitErrorNT (net.js:1277:8)\n    at _combinedTickCallback (internal/process/next_tick.js:80:11)\n    at process._tickCallback (internal/process/next_tick.js:104:9)","levelPath":"fatal/request"},"msg":"Config Request failure for \"http://localhost/w/api.php\": Error: connect ECONNREFUSED 127.0.0.1:80","time":"2017-09-05T20:29:40.044Z","v":0}
{"name":"parsoid","hostname":"mediawiki-vagrant","pid":1192,"level":40,"logType":"warn/api/econnrefused","wiki":"localhost","title":"Book","oldId":null,"reqId":null,"userAgent":"curl/7.38.0","msg":"Failed API request, {\"error\":{\"code\":\"ECONNREFUSED\",\"errno\":\"ECONNREFUSED\",\"syscall\":\"connect\",\"address\":\"127.0.0.1\",\"port\":80},\"retries-remaining\":1}","longMsg":"Failed API request,\n{\"error\":{\"code\":\"ECONNREFUSED\",\"errno\":\"ECONNREFUSED\",\"syscall\":\"connect\",\"address\":\"127.0.0.1\",\"port\":80},\"retries-remaining\":1}","levelPath":"warn/api/econnrefused","time":"2017-09-05T22:44:05.503Z","v":0}

The correct MW API URL is http://localhost:8080/w/api.php. I tried to fix this issue for installations with non-default ports with https://gerrit.wikimedia.org/r/#/c/375933/ , maybe that broke it somehow for the default port? /vagrant/srv/parsoid/config.vagrant.yaml has http://localhost:8080/w/api.php for the MW API URL though, I am not sure where the :80 request is coming from.

So apparently ::forwarded_port and ::port_fragment are the same on normal Vagrant but different in Labs, and that breaks a bunch of things, including RESTBase. Patched puppet locally, tracking the writing of a proper fix in T175100.

OCG works now (had to restart the host first due to node hanging up).

Thanks, @Tgr!

Is there anything in particular you need from developers with respect to "code review" or can we move this to sign off ?

Will git auto update here or do we have to do it manually?

I haven't setup any auto update for the sake of stable and predictable behavior. We'll have to maintain the instance manually.

Prior to resolving this task, would it be worth documenting what the had to do to set up the staging environment on a wiki (next to Setting up a staging environment, perhaps?)

I'd rather we don't as that document will get outdated without active maintenance. Things are already well documentaed at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS.

ovasileva moved this task from Backlog to Current Sprint on the Proton board.Sep 6 2017, 6:57 PM

ovasileva moved this task from Current Sprint to Resolved on the Proton board.

In T174430#3584540, @Jdlrobson wrote:

Will git auto update here or do we have to do it manually?

vagrant git-update will work the same way as on any other normal vagrant box (ie. it will do the update but will skip repos which are not on master or have uncommitted changes; for repos which are on master but have diverged, it will try a rebase and might need manual cleanup).

Note that Collection behaves somewhat erratically on Vagrant (or probably any kind of local installation): you get error messages when trying to add pages, but usually they get added anyway. I didn't want to spend time on tracking that down as it's going to be rewritten anyway, but in any case it's not specific to the staging environment.

Change 376487 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/vagrant@master] Add ElectronVirtualRestService config to offline role

https://gerrit.wikimedia.org/r/376487

gerritbot added a project: Patch-For-Review.Sep 7 2017, 9:23 AM

In T174430#3585809, @bmansurov wrote:

I'd rather we don't as that document will get outdated without active maintenance. Things are already well documented at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS.

I agree that the general process of setting up labs instance is well documented there as well as a few gotchas – indeed, it's linked from the MobileFrontend-specific doc.

I'm not sure if there are gotchas specific to the offline role that might be worth writing down. If you think not, then fine.

Jdlrobson removed a project: Patch-For-Review.Sep 7 2017, 2:55 PM

I think we can try and link this ticket at most. Even the offline role is changing, so things may get old and useless later.

Looks like Sam might be signing this off? Please re-assign if that's not the case.

Most problems that came up should be fixed in Vagrant, not documented.

Maybe the one thing worth mentioning is that you can use production RESTBase (for comparing PDF output to production, for example) with a setting like this:

$wgVirtualRestConfig['modules']['restbase'] = [
        'url' => 'https://en.wikipedia.org/api/rest_v1/',
        'domain' => 'en.wikipedia.org',
        'forwardCookies' => false,
        'fixedUrl' => true,
] + $wgVirtualRestConfig['modules']['restbase'];

(The local MW API will be used for sections etc. so you'll need to have the same wikitext locally, but the HTML will be fetched from enwiki.)

phuedx updated the task description. (Show Details)Sep 8 2017, 10:14 AM

phuedx updated the task description. (Show Details)

As well as verifying that I could download a page as a one or two column PDF, I was bold and created mw:Reading/Web/Setting_up_a_Proton_staging_environment. It's simple and isn't tightly coupled to any MWV issue that y'all have encountered while working on this task.

🎉🎉🎉

Change 376487 merged by jenkins-bot:
[mediawiki/vagrant@master] Add ElectronVirtualRestService config to offline role