Page MenuHomePhabricator

Fix or clarify Proton Puppeteer sandboxing and SSL settings
Closed, ResolvedPublic3 Estimated Story Points

Description

Per T177765#4867361:

  • Puppeteer should preferably use the built-in sandbox (ie. do not set --no-sandbox, -disable-setuid-sandbox options). If that's not possible, the reasons should be documented. If it's not possible and other sandboxing is possible (e.g. firejail), we should look into that.
  • Puppeteer should properly verify SSL certs; the ignoreHTTPSErrors probably disables that (if not, a reassuring comment would be good). If that's not possible, the reasons should be documented.

Acceptance criteria

  • Document the use of --ignoreHTTPSErrors flag
  • Document the use of --no-sandbox flag and the firejail sandboxing in the project page or wikitech docs

Event Timeline

Is this running in a container? In CI we have to set --no-sandbox because Chromium's sandbox doesn't work inside docker containers.

AIUI the service runner running Proton is firejailed. Is that correct @pmiazga, @Pchelolo?

AIUI the service runner running Proton is firejailed. Is that correct @pmiazga, @Pchelolo?

That is correct @phuedx

I think I know why we have ignoreHTTPSErrors and it shouldn't be necessary anymore. I'll check all params and I'll get back to you.
@Legoktm - the service itself doesn't run in a docker container, but the build process is run via docker, that may be the issue.

Sandboxing in Chromium is confusing. The SUID sandbox is apparently on its way out (although this other place lists it as maintianed)? The sandbox disabled with --no-sandbox seems Windows only? But there is also the namespace sandbox which is the successor of SUID sandbox, but has no disable switch? And the [[https://chromium.googlesource.com/chromium/src/+/master/docs/linux_sandboxing.md#the-sandbox-1|Seccomp-BPF]] sandbox, which does not have an off switch either.

No idea where that leaves us.

Jhernandez subscribed.

So, for this one, lets:

  • Remove the ignoreHTTPSErrors flag
  • Document the use of --no-sandbox flag and the firejail sandboxing in the project page or wikitech docs
pmiazga updated the task description. (Show Details)

We cannot remove the --ignoreHTTPSErrors flag. It was added to the config https://github.com/wikimedia/mediawiki-services-chromium-render-deploy/commit/17fc7bb4cda2e1b408d6289c27685624d35e8714 because:

We use a self-signed certificate for our domains (since the CA is our Puppet). However, that results in Chromium HTTPS errors. Given that Proton cannot communicate with the outside world, it is safe to use the ignoreHTTPSErrors configuration flag since requesting HTML from our appservers is all the service does.

Instead of removing the IgnoreHTTPSErrors flag, I'll document why it is used.

Done: https://wikitech.wikimedia.org/wiki/Proton#Puppeteer_configuration

@Tgr @Jhernandez please check the puppeteer config documentation and let me know if it meets your expectations.

... https://wikitech.wikimedia.org/wiki/Proton#Puppeteer_configuration

... please check the puppeteer config documentation and let me know if it meets your expectations.

Moving to our kanban for proof-reading by the devs. Please have a look and provide feedback.

We use a self-signed certificate for our domains (since the CA is our Puppet).

Which sites are those? Public Wikimedia sites use certificates signed by GlobalSign. Beta uses LetsEncrypt certificates which are technically also not self-signed.

(Also if we use self-signed certificates somewhere we could just import those to the cert store of the Chromium instance. Although if Proton will connect to the websites internally, maybe not worth the effort as HTTPS encryption does not play much of a security role anyway. Is that the case for Proton instances in the passive DC, though?)

@Tgr - Public Wikimedia sites and Beta certs are set for public domains. Proton render used to access Wikipedia servers using internal network names (as it was inside internal network). The internal servers use the CA generated by Puppet.
Switching between HTTP/HTTPS modes is available via config (the config file has the URL template used to build the article URL).

I checked, and it looks like it's difficult (most probably not possible). Puppeteer doesn't allow to modify browser certificates - if it did - it would be a big security loophole.

@mobrovac can provide more details on this problem.

I see. Is there a point in using internal domain names, though? All the resources linked in the page (including some dynamic ones like ResourceLoader URLs) will use the public ones anyway.

(In theory you can programmatically manage certificates through the NSS store. But yeah probably not worth spending a lot of time on.)

I'll ping @mobrovac once again, as he decided to use internal domain names (previously we were using the public domains).

We are currently using the projects' FQDNs in MW API requests becasue this was the only way to get the CSS we needed for the printable version (not sure why that is the case, but that's a problem for another day). So, for the time being, SSL certificates are being honoured by Chromium.

That said, we would like to use the internal domain as this is the standard way to contact the appservers in production.

Tgr assigned this task to pmiazga.

I'd guess the CSS problem is because CSS (at least some of it) will be loaded from the public domain, whatever URL you use to fetch the page, and then you end up with different domains for the HTML page and the CSS stylesheet, and get in trouble with CSP or CORS or something similar that cares about same-origin restrictions. IMO just using the public URL is less trouble (and means you can also make use of Varnish, although given the URL structure used by Proton I'm not sure that would happen, and with PDF rendering times it probably won't make much difference either way).

That said I think the security impact is minimal here (worst case, an attacker can do a DNS poisoning attack and replace the page HTML, but they would probably have to poison the WMF's own DNS cache for that, and even if they pull it off, Proton should be able to handle malicious content safely) so we can close this task and figure out whether to use the internal hostnames (and consequently whether we need to disable SSL checks in the long run) later.