Page MenuHomePhabricator

[2 hrs] Decide on handling system updates for Proton
Closed, ResolvedPublic

Description

Proton is a headless Chromium process remote-controlled by Puppeteer. Puppeteer is fairly sensitive about the Chromium version being used. The recommended method is via an npm install hook in Puppeteer, which downloads the Chromium version it likes. We probably don't want to do that as it would it invisible whether we use an outdated/insecure Chromium version. We probably don't want to use the OS-native Chromium version either as that would mean the service could break without warning on any OS upgrade. So presumably pin the Chromium package version (that might make it outdated but at least it's visible) and have some process of updating the pinning every time Puppeteer itself is updated? Sounds a bit painful.

Also we need to document the process of testing Proton with a new Chromium version on Beta. (Mainly thinking of the use case where Chromium has a security vulnerability / rendering issue / whatever that wants us to use a newer version than what Puppeteer is specifically documented to work with, and we need to make sure that doesn't break anything.)

Event Timeline

Tgr created this task.Jan 10 2019, 2:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 10 2019, 2:35 AM
herron triaged this task as Normal priority.Jan 10 2019, 10:21 PM
herron added a project: Security-Team.
herron added a subscriber: MoritzMuehlenhoff.
herron added a subscriber: Volans.

@phuedx and I looked at https://www.mediawiki.org/wiki/Proton#Technical_documents and https://wikitech.wikimedia.org/wiki/Proton and couldn't find docs for this.

We decided @pmiazga will document this and we will review the docs for resolving 👍

ovasileva renamed this task from Decide on handling system updates for Proton to [2 hrs] Decide on handling system updates for Proton.Jan 15 2019, 5:54 PM

Done : documented how to perform updates and wrote down why we do it this way: https://wikitech.wikimedia.org/wiki/Proton#Updating_Puppeteer_&_Chromium

@Tgr @Jhernandez - could you verify the documentation and sign of the task?

...documented how to perform updates and wrote down why we do it this way: https://wikitech.wikimedia.org/wiki/Proton#Updating_Puppeteer_&_Chromium
... could you verify the documentation and sign of the task?

Moving into our kanban board to get proof-reading from devs. Please have a look and provide feedback.

Jhernandez removed Jhernandez as the assignee of this task.Jan 23 2019, 5:10 PM
Tgr added a comment.EditedJan 24 2019, 4:48 PM

Thanks! It's good to have the status quo precisely documented (and the insights on why Chromium cannot be bundled are valuable). I think this is still worth looking into - AIUI with the current setup Puppet will upgrade Chromium whenever Debian updates their package, so the service could break without any action from our side. Pinning has its own set of problems (it would have to be done in Puppet so it's hard to coordinate a change of Puppeteer version with a change of Chromium version); maybe there is a better solution I couldn't think of.

AIUI with the current setup Puppet will upgrade Chromium whenever Debian updates their package, so the service could break without any action from our side.

Packages in production are not upgraded by Puppet, that's handled by SRE. Specifically for Proton, when there's a new Chromium release in Debian, I test the new release in deployment-prep and only upgrade the chromium packages on proton* after that.

Tgr added a comment.Jan 24 2019, 7:35 PM

@MoritzMuehlenhoff thanks, that's good to know. How would the process look when the new Chromium release is not compatible with the Puppeteer version used in Proton? Would there be some time frame within which Proton is expected to be updated, or having an outdated Chromium version for a longer time is not really an issue?

What would be the process for upgrading when the new Chromium is not compatible with the old Puppeteer version and the new Puppeteer version is not compatible with the old Chromium (so the Proton code deployment and the Chromium package update would have to be more or less simultaneous)?

@Tgr so far we were bumping Puppeteer version and usually everything works. There was one problem with headers (chromium wasn't sending the headers we're asking for) but most probably that was the puppeteer lib issue in given version, not the conclict between two different versions. After we bumped to the newer release problem went away. We started chromium-render development from Puppeteer 0.11.0 (https://github.com/wikimedia/mediawiki-services-chromium-render/commit/d02e5a57ec1e986f0992edaf6b8c8169d13b5203#diff-b9cfc7f2cdf78a7f4b91a753d10865a2) and so far update process didn't cause any troubles (currently we're on 1.9.0)

The easiest way to verify that everything still works properly is to do two steps:
a) run unit tests. There is an integration test that tries to generate a PDF from English Wikipedia, if it works - most probably everything is fine.
b) the second test has to be manual, someone has to generate the PDF and verify that it looks correct , we could add some extra checks to the integration tests mentioned in a), for now it only verifies that http server returns 200, pdf mime type and body is not empty. Additionally, we can use some PDF nodejs library and try to open the buffer to check that generated response is a correct PDF file.

hashar added a subscriber: hashar.Feb 7 2019, 10:58 AM

TLDR: in the CI job, puppeteer does not download chromium but relies on the Debian package. The jobs execute in Docker containers which freeze the chromium version. We might want to use the CI jobs to validate the package is upgrade is working (== it passes tests).


When the chromium-render service has been setup (T179552 T186748) we have made CI to prevent puppeteer from downloading Chrome from Google. The Docker container that runs the tests does have the Chromium Debian packages that comes from Debian. I had pointed out at the time that in production we would not download a random binary but would most probably use the chromium package (and indeed that is the path we have took).

The way the download is disabled in CI is by injecting PUPPETEER_SKIP_CHROMIUM_DOWNLOAD='true' to jobs triggered for the Gerrit repository mediawiki/services/chromium-render. Specially in integration/config.git:

zuul/parameter_functions.py
# Prevent puppeteer from downloading Chromium, we use the Debian package
# instead.  T179552 T186748
if params['ZUUL_PROJECT'].startswith('mediawiki/services/chromium-render'):
    params['PUPPETEER_SKIP_CHROMIUM_DOWNLOAD'] = 'true'

We then trigger the following jobs:

zuul/layout.yaml
- name: mediawiki/services/chromium-render
  test:
    - chromium-render-npm-browser-node-6-docker
  gate-and-submit:
    - chromium-render-npm-browser-node-6-docker

- name: mediawiki/services/chromium-render/deploy
  experimental:
    - chromium-render-deploy-npm-node-6-docker

The jobs are currently tied to the Docker container docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1:

jjb/mediawiki-services.yaml
- project:
    name: chromium-render
    jobs:
     - '{name}-npm-browser-node-6-docker':
         docker_image_var: docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1
     - '{name}-deploy-npm-node-6-docker':
         docker_image_var: docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1

Which currently has Chromium 69 (when Stretch latest is currently 71):

$ docker run --rm -it --entrypoint=chromium docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1 --version
Chromium 69.0.3497.92 built on Debian 9.5, running on Debian 9.1

All that to say that we can rebuild that container to get the latest Chromium then craft an additional job that has the newest version and trigger it alongside as a non voting job. Then when it works, disable that job and upgrade the voting job.


NOTE: All that being said, maybe we should migrate the service to kubernetes and use the pipeline T198901. The test would use the latest version of Chromium, run tests and if that passes the image get published to the registry which we can then use on production with some high confidence it is working.

I've removed this from our (Readers Web's) kanban board as Proton has been handed over to Readers Infrastructure.

@MoritzMuehlenhoff thanks, that's good to know. How would the process look when the new Chromium release is not compatible with the Puppeteer version used in Proton? Would there be some time frame within which Proton is expected to be updated, or having an outdated Chromium version for a longer time is not really an issue?
What would be the process for upgrading when the new Chromium is not compatible with the old Puppeteer version and the new Puppeteer version is not compatible with the old Chromium (so the Proton code deployment and the Chromium package update would have to be more or less simultaneous)?

Sorry, missed to reply because of Allhands backlog, but turns out we actually have that case now: Chromium 72 as released by Debian yesterday fails to work on the Proton hosts. I've opened https://phabricator.wikimedia.org/T216493

Of course, I'll withhold upgrading the production proton hosts for now, but we should fix this ASAP (probably a Puppeteer upgrade is needed).

ovasileva moved this task from Triage to Backlog on the Proton board.Feb 22 2019, 3:04 PM
MSantos closed this task as Resolved.Jul 25 2019, 4:10 PM
MSantos claimed this task.
MSantos added a subscriber: MSantos.

So, the current solution is to keep using Debian stable packages and stay vigilant with version compatibility between puppeteer and chromium.

sbassett moved this task from Backlog to Done on the Security-Team board.Fri, Aug 30, 4:12 PM