Page MenuHomePhabricator
Create Task
Maniphest T198461

Proton cannot assume the requests are for {lang}.wikipedia.org
Closed, ResolvedPublic
Actions

Description

The Proton service assumes that the requests it receives are for wikipedia pages only. However, its aim is to replace the Electron rendering service, which is enabled for all projects. Therefore, Proton cannot assume that the requests for PDFs will be limited to Wikipedia.

Details

SubjectRepoBranchLines +/-
Config: Improve the MW request templatemediawiki/services/chromium-render/deploymaster+8 -5
Tighten the MW requestmediawiki/services/chromium-rendermaster+67 -35
service::node: Expose the MW appservers' host to modulesoperations/puppetproduction+10 -11
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedovasilevaT181079 [GOAL] Provide an expanded reading experience by improving the ways that users can download articles of interest for later consumption
 ResolvedNoneT181084 [EPIC] Deploy the mediawiki-services-chromium-render service (Proton)
 ResolvedBawolffT177765 Security review of mediawiki-services-chromium-render
 ResolvedphuedxT178071 [EPIC] Complete the First Deployment tutorial/checklist for the trial render service
 Resolved• mobrovacT178168 Scap3ize the deploy repository
 Resolved• mobrovacT178172 Create Puppet role and profile
 ResolvedphuedxT181118 [EPIC] Build a Chromium-based PDF renderer service
 ResolvedJdrewniakT179552 Set up Jenkins for chromium-render repository
 ResolvedphuedxT180601 Add logging to the mediawiki-services-chromium-render service
 ResolvedphuedxT180626 [Spike 8hr] How should we limit resources used by chromium render service?
 ResolvedphuedxT178278 Performance test the service
 ResolvedphuedxT178501 Limit resources used by Chromium in order to make the chromium-render service ready for production
 DeclinedNoneT178186 Update Node.js version/dependencies in package.json
 Resolved• bmansurovT176627 Trial replacing Electron with headless Chromium in the render service
 Resolved• mobrovacT180800 Update to service-template-node v0.5.3
 ResolvedovasilevaT180604 Chromium render keeps tasks in the queue even browser connection is lost
 InvalidNoneT183161 Performance test books on chromium rendering service
 ResolvedJdlrobsonT184609 [4hrs] Document the service
 ResolvedphuedxT189307 Instrument the Proton service to match mediawiki-services-electron-render
 ResolvedphuedxT181097 Update puppeteer regularly
 Resolved• holger.knustT210651 Switch all PDF render traffic to new Proton service
 ResolvedTgrT213362 Limit what URLs Proton can access
 ResolvedjijikiT226675 Undeploy electron service from WMF production
 ResolvedMSantosT226677 Expand RESTBase public PDF rendering API to support additional options supported by Proton
 Resolved• mobrovacT197862 Increase the CPU count for proton[12]00[12]
 ResolvedovasilevaT210793 Verify that PDFs generated by Proton service are correct
 ResolvedphuedxT210652 Handoff Proton service to Reading Infrastructure
 DeclinedpmiazgaT211375 Rename chromium-render gerrit project to Proton
 ResolvedDzahnT211382 Requesting access to Proton for pmiazga, bearND, Mholloway, MSantos, Tgr
 ResolvedpmiazgaT213363 Fix or clarify Proton Puppeteer sandboxing and SSL settings
 Duplicate• holger.knustT215557 Remove electron pdf endpoint backend in RESTBase
 ResolvedMSantosT213366 [2 hrs] Decide on handling system updates for Proton
 ResolvedTgrT217724 Investigate 2019-03-01 Proton incident
 ResolvedJohanT220648 Communications plan for Electron -> Proton switchover
 OpenNoneT158181 Aim for workflow equivalence for MediaWiki on desktop and mobile web
 ResolvedovasilevaT163472 [EPIC] Provide a way to download articles in PDF on the mobile website
 InvalidNoneT181513 Prepare for deploy of chromium rendering service and usage on mobile (traffic)
 Resolved• mobrovacT186748 [EPIC] New service request: chromium-render/deploy
 Resolved• mobrovacT198461 Proton cannot assume the requests are for {lang}.wikipedia.org

Event Timeline

• mobrovac triaged this task as High priority.Jun 29 2018, 9:35 AM
• mobrovac created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 29 2018, 9:35 AM
• mobrovac added a parent task: T186748: [EPIC] New service request: chromium-render/deploy.Jun 29 2018, 9:36 AM
pmiazga added a comment.Jun 29 2018, 9:48 AM

Maybe instead of doing a restbase check (calling /page/title/{TITLE}, first we can do a HEAD call to the requested URL and check the http response. if it's 200 proceed with the queue, otherwise reject the job immediately? With that approach, we will be able to handle all possible projects.

• mobrovac added a comment.Jun 29 2018, 4:14 PM

The RESTBase call is not wikipedia-specific, it supports all the wikis where RESTBase is enabled. The problem is in this fragment, which assumes a specific format for the domain.

pmiazga added a subscriber: • Pchelolo.Jun 29 2018, 7:00 PM

That part is done to handle the mobile domains (adding .m part). There is no nice way to retrieve the mobile domain for a given wiki. We already did some research on this case, please check:

There is also some small conversation in gerrit patches

Krenair subscribed.Jun 29 2018, 7:39 PM
In T198461#4326202, @pmiazga wrote:

That part is done to handle the mobile domains (adding .m part). There is no nice way to retrieve the mobile domain for a given wiki.

have it defined in config?

pmiazga added a comment.Jun 29 2018, 8:55 PM

How to build the mobile domain in the Proton service -> yes, the template is stored in the config: https://github.com/wikimedia/mediawiki-services-chromium-render/blob/master/config.dev.yaml#L66

• mobrovac claimed this task.Jul 2 2018, 2:51 PM
• mobrovac edited projects, added Services (doing); removed Services (blocked).

We actually also need to do this because of security implications: we want to restrict the service to only be allowed to talk to our production MW servers directly.

In T198461#4326202, @pmiazga wrote:

That part is done to handle the mobile domains (adding .m part). There is no nice way to retrieve the mobile domain for a given wiki. We already did some research on this case, please check:

Actually, there is. As far as our production servers are considered, .m. domains do not exist, only their desktop counterparts. In other words, both en.wp.org/wiki/Title and en.m.wp.org/wiki/Title get interpreted in the same way (as being for the en.wp.org domain). I will try to add the logic to Proton and change the MW request template to reflect that.

gerritbot subscribed.Jul 2 2018, 3:30 PM

Change 443444 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] service::node: Expose the MW appservers' host to modules

https://gerrit.wikimedia.org/r/443444

gerritbot added a project: Patch-For-Review.Jul 2 2018, 3:30 PM
gerritbot added a comment.Jul 2 2018, 4:55 PM

Change 443465 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/chromium-render@master] Tighten the MW request

https://gerrit.wikimedia.org/r/443465

gerritbot added a comment.Jul 2 2018, 5:06 PM

Change 443468 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/chromium-render/deploy@master] Config: Improve the MW request template

https://gerrit.wikimedia.org/r/443468

• mobrovac mentioned this in rESCDef8a54e9a81f: Config: Improve the MW request template.Jul 2 2018, 5:06 PM
• mobrovac added a comment.Jul 2 2018, 5:09 PM

The three patches above should get us what we want, as they allow us to restrict the hosts we issue requests to in production (these being only the MW app servers), all the while allowing us to fetch both desktop and mobile views of pages.

• mobrovac mentioned this in rMSCRfb6f8daf55a6: Tighten the MW request.Jul 2 2018, 5:22 PM
• mobrovac mentioned this in rMSCRa3ee7f28c443: Tighten the MW request.
• mobrovac mentioned this in rMSCRc08ced501ad8: Tighten the MW request.Jul 3 2018, 11:20 AM
gerritbot added a comment.Jul 4 2018, 6:06 AM

Change 443444 merged by Giuseppe Lavagetto:
[operations/puppet@production] service::node: Expose the MW appservers' host to modules

https://gerrit.wikimedia.org/r/443444

• mobrovac mentioned this in rMSCR0d76a155d7fc: Tighten the MW request.Jul 16 2018, 5:56 PM
gerritbot added a comment.Jul 26 2018, 5:13 PM

Change 443465 merged by Mobrovac:
[mediawiki/services/chromium-render@master] Tighten the MW request

https://gerrit.wikimedia.org/r/443465

gerritbot added a comment.Jul 26 2018, 7:54 PM

Change 443468 merged by Mobrovac:
[mediawiki/services/chromium-render/deploy@master] Config: Improve the MW request template

https://gerrit.wikimedia.org/r/443468

Stashbot subscribed.Jul 26 2018, 7:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T19:58:28Z] <mobrovac@deploy1001> Started deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461

Stashbot added a comment.Jul 26 2018, 7:59 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-26T19:59:02Z] <mobrovac@deploy1001> Finished deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461 (duration: 00m 33s)

• mobrovac closed this task as Resolved.Jul 26 2018, 7:59 PM
• mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (doing).

Deployed, resolving.

• mobrovac mentioned this in rESCD883cacd20762: Config: Improve the MW request template.Jul 26 2018, 10:19 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL