@phuedx and I looked at https://www.mediawiki.org/wiki/Proton#Technical_documents and https://wikitech.wikimedia.org/wiki/Proton and couldn't find docs for this.

We decided @pmiazga will document this and we will review the docs for resolving 👍

Done : documented how to perform updates and wrote down why we do it this way: https://wikitech.wikimedia.org/wiki/Proton#Updating_Puppeteer_&_Chromium

@Tgr @Jhernandez - could you verify the documentation and sign of the task?

In T213366#4902530, @pmiazga wrote:

...documented how to perform updates and wrote down why we do it this way: https://wikitech.wikimedia.org/wiki/Proton#Updating_Puppeteer_&_Chromium
... could you verify the documentation and sign of the task?

Moving into our kanban board to get proof-reading from devs. Please have a look and provide feedback.

Thanks! It's good to have the status quo precisely documented (and the insights on why Chromium cannot be bundled are valuable). I think this is still worth looking into - AIUI with the current setup Puppet will upgrade Chromium whenever Debian updates their package, so the service could break without any action from our side. Pinning has its own set of problems (it would have to be done in Puppet so it's hard to coordinate a change of Puppeteer version with a change of Chromium version); maybe there is a better solution I couldn't think of.

In T213366#4906157, @Tgr wrote:

AIUI with the current setup Puppet will upgrade Chromium whenever Debian updates their package, so the service could break without any action from our side.

Packages in production are not upgraded by Puppet, that's handled by SRE. Specifically for Proton, when there's a new Chromium release in Debian, I test the new release in deployment-prep and only upgrade the chromium packages on proton* after that.

@MoritzMuehlenhoff thanks, that's good to know. How would the process look when the new Chromium release is not compatible with the Puppeteer version used in Proton? Would there be some time frame within which Proton is expected to be updated, or having an outdated Chromium version for a longer time is not really an issue?

What would be the process for upgrading when the new Chromium is not compatible with the old Puppeteer version and the new Puppeteer version is not compatible with the old Chromium (so the Proton code deployment and the Chromium package update would have to be more or less simultaneous)?

@Tgr so far we were bumping Puppeteer version and usually everything works. There was one problem with headers (chromium wasn't sending the headers we're asking for) but most probably that was the puppeteer lib issue in given version, not the conclict between two different versions. After we bumped to the newer release problem went away. We started chromium-render development from Puppeteer 0.11.0 (https://github.com/wikimedia/mediawiki-services-chromium-render/commit/d02e5a57ec1e986f0992edaf6b8c8169d13b5203#diff-b9cfc7f2cdf78a7f4b91a753d10865a2) and so far update process didn't cause any troubles (currently we're on 1.9.0)

The easiest way to verify that everything still works properly is to do two steps:
a) run unit tests. There is an integration test that tries to generate a PDF from English Wikipedia, if it works - most probably everything is fine.
b) the second test has to be manual, someone has to generate the PDF and verify that it looks correct , we could add some extra checks to the integration tests mentioned in a), for now it only verifies that http server returns 200, pdf mime type and body is not empty. Additionally, we can use some PDF nodejs library and try to open the buffer to check that generated response is a correct PDF file.

TLDR: in the CI job, puppeteer does not download chromium but relies on the Debian package. The jobs execute in Docker containers which freeze the chromium version. We might want to use the CI jobs to validate the package is upgrade is working (== it passes tests).

When the chromium-render service has been setup (T179552 T186748) we have made CI to prevent puppeteer from downloading Chrome from Google. The Docker container that runs the tests does have the Chromium Debian packages that comes from Debian. I had pointed out at the time that in production we would not download a random binary but would most probably use the chromium package (and indeed that is the path we have took).

The way the download is disabled in CI is by injecting PUPPETEER_SKIP_CHROMIUM_DOWNLOAD='true' to jobs triggered for the Gerrit repository mediawiki/services/chromium-render. Specially in integration/config.git:

# Prevent puppeteer from downloading Chromium, we use the Debian package
# instead.  T179552 T186748
if params['ZUUL_PROJECT'].startswith('mediawiki/services/chromium-render'):
    params['PUPPETEER_SKIP_CHROMIUM_DOWNLOAD'] = 'true'

We then trigger the following jobs:

- name: mediawiki/services/chromium-render
  test:
    - chromium-render-npm-browser-node-6-docker
  gate-and-submit:
    - chromium-render-npm-browser-node-6-docker

- name: mediawiki/services/chromium-render/deploy
  experimental:
    - chromium-render-deploy-npm-node-6-docker

The jobs are currently tied to the Docker container docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1:

- project:
    name: chromium-render
    jobs:
     - '{name}-npm-browser-node-6-docker':
         docker_image_var: docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1
     - '{name}-deploy-npm-node-6-docker':
         docker_image_var: docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1

Which currently has Chromium 69 (when Stretch latest is currently 71):

$ docker run --rm -it --entrypoint=chromium docker-registry.wikimedia.org/releng/npm-browser-test:0.3.1 --version
Chromium 69.0.3497.92 built on Debian 9.5, running on Debian 9.1

All that to say that we can rebuild that container to get the latest Chromium then craft an additional job that has the newest version and trigger it alongside as a non voting job. Then when it works, disable that job and upgrade the voting job.

I've removed this from our (Readers Web's) kanban board as Proton has been handed over to Readers Infrastructure.

In T213366#4906913, @Tgr wrote:

@MoritzMuehlenhoff thanks, that's good to know. How would the process look when the new Chromium release is not compatible with the Puppeteer version used in Proton? Would there be some time frame within which Proton is expected to be updated, or having an outdated Chromium version for a longer time is not really an issue?

What would be the process for upgrading when the new Chromium is not compatible with the old Puppeteer version and the new Puppeteer version is not compatible with the old Chromium (so the Proton code deployment and the Chromium package update would have to be more or less simultaneous)?

Sorry, missed to reply because of Allhands backlog, but turns out we actually have that case now: Chromium 72 as released by Debian yesterday fails to work on the Proton hosts. I've opened https://phabricator.wikimedia.org/T216493

Of course, I'll withhold upgrading the production proton hosts for now, but we should fix this ASAP (probably a Puppeteer upgrade is needed).