Page MenuHomePhabricator
Create Task
Maniphest T226078

releng/quibble-fresnel container requires Firefox to be installed in order to launch Chromium
Closed, ResolvedPublic
Actions

Description

Spotted by @Jdforrester-WMF , docker-registry.wikimedia.org/releng/quibble-fresnel:0.0.31-5 fails to start Chromium in headless mode. We thus had to add firefox as a dependency which is a terrible hack:

https://gerrit.wikimedia.org/r/#/c/integration/config/+/517121/3/dockerfiles/quibble-fresnel/Dockerfile.template

From digging in Jenkins builds, a couple builds failed:

https://integration.wikimedia.org/ci/job/mediawiki-fresnel-patch-docker/5055/
https://integration.wikimedia.org/ci/job/mediawiki-fresnel-patch-docker/5056/

INFO:test.commands:mediawiki-fresnel-patch
+ export FRESNEL_DIR=/workspace/log/fresnel_records
+ FRESNEL_DIR=/workspace/log/fresnel_records
+ fresnel record after
Error: Failed to launch chrome!
/opt/fresnel/node_modules/fresnel/node_modules/puppeteer/.local-chromium/linux-624492/chrome-linux/chrome: error while loading shared libraries: libatk-bridge-2.0.so.0: cannot open shared object file: No such file or directory


TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md

    at onClose (/opt/fresnel/node_modules/fresnel/node_modules/puppeteer/lib/Launcher.js:360:14)
    at Interface.helper.addEventListener (/opt/fresnel/node_modules/fresnel/node_modules/puppeteer/lib/Launcher.js:349:50)
    at Interface.emit (events.js:194:15)
    at Interface.close (readline.js:379:8)
    at Socket.onend (readline.js:157:10)
    at Socket.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
$ docker run --rm -it --entrypoint=apt-cache docker-registry.wikimedia.org/releng/quibble-fresnel:0.0.31-5 policy chromium
chromium:
  Installed: 73.0.3683.75-1~deb9u1
  Candidate: 73.0.3683.75-1~deb9u1
  Version table:
 *** 73.0.3683.75-1~deb9u1 100
        100 /var/lib/dpkg/status

Details

SubjectRepoBranchLines +/-
fresnel: Restore comment about firefox-esrintegration/configmaster+9 -3
Customize query in gerrit

Related Objects

Event Timeline

hashar created this task.Jun 19 2019, 8:13 AM
Restricted Application added a project: Performance-Team. · View Herald TranscriptJun 19 2019, 8:13 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar updated the task description. (Show Details)Jun 19 2019, 8:17 AM
hashar added a subscriber: Krinkle.Jun 19 2019, 8:28 AM

The reason is that it uses puppeteer which download chromium but obviously fail to install the dependencies :-\ There are a few ways to solve that:

A) Replace puppeteer with puppeteer-core which does not download Chromium but expect it to be available in the environment.

B) Pass an environment variable to prevent downloading Chromium PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1. We might also want to pass it the executable location with PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium. That is the path we took for:

C) rely on the chromium downloaded from Puppeteer, we would thus have to install chromium dependencies explicitly, and stop installing Chromium entirely. The trouble is that we will need to keep track of the proper dependencies and keep updating them whenever a new version of Chromium is downloaded by puppeteer.

Side note: the root cause of the failure is the container has chromium 73.0.3683.75-1~deb9u1 which is linked to libatk-1.0.so.0, when the newer version of Chromium requires libatk-bridge2.0-0. Installing Firefox bring in that library and fix it, but it is a hack really.

Krinkle added a comment.Jun 19 2019, 1:35 PM

This is not new.

When I wrote the Dockerfile for Fresnel 4 months ago (1b37556b56d) this was already thought about and resolved.

It looks new because last week in 80c2a544ab0/b0e50b2d, James accidentally removed these apt_install commands and their doc comment. The apt commands were added back in 14f9f28e0, and I've now added back the doc comment as well.

In T226078#5267830, @hashar wrote:

A) Replace puppeteer with puppeteer-core which does not download Chromium but expect it to be available in the environment.

This is not acceptable in terms of maintenance cost and loss in compatibility. Upstream Puppeteer and upstream Chromium don't do cross-version support. Puppeteer works with the version it is bundled with. That's how it is supported, released and distributed, and that's how Fresnel uses it.

Changing this means Fresnel may become unusable for developers locally, which makes development harder, and diminishes a lot of its value. It also means that it can break at random when I'm not around when Docker images are rebuilt for unrelated reasons (e.g. sec patches, or Quibble/PHP upgrades).

Puppeteer wraps Chromium, and Fresnel wraps Puppeteer. This install is currently deterministic and controlled in the Dockerfile. Cheap in maintenance, high in stability, and provides maximum usability for all users and developers interacting with it.

B) Pass an environment variable to prevent downloading Chromium PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1. We might also want to pass it the executable location with PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium. That is the path we took for: [..]

Same problems as above. But, if Fresnel were a production service and if it were deployed through a test-approved Docker-based pipeline, then this would be worth investing it. We would then stop support for Fresnel from npm, and instead require developers to install it as a Docker image locally as well.

But, it is none of those things right now, and doing that adds maintenance cost with no benefit for us.

C) rely on the chromium downloaded from Puppeteer, we would thus have to install chromium dependencies explicitly, and stop installing Chromium entirely. The trouble is that we will need to keep track of the proper dependencies and keep updating them whenever a new version of Chromium is downloaded by puppeteer.

This is what I choose 4 months ago and what we are still doing. The version of Chromium is controlled by Fresnel. When updating Fresnel version in the quibble-fresnel image, I test it locally, and if the updated version of Fresnel (incl Puppeteer/Chromium) needs additional packages, I make sure those are installed. For now, the proxy of chromium and firefox-esr covers everything it needs and matches how we use Chromium elsewhere in WMF CI.

For example, when we run Selenium WDIO tests and QUnit tests, we also use Headless Chromium. And the only reason that works right now is because firefox-esr is installed in those images as well.

If and when this assumption stops being true, then on the next Frensel update in quibble-fresnel after that, I'll naturally add the needed packages.

gerritbot added a comment.Jun 19 2019, 1:35 PM

Change 517860 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] fresnel: Restore comment about firefox-esr

https://gerrit.wikimedia.org/r/517860

gerritbot added a project: Patch-For-Review.Jun 19 2019, 1:35 PM
gerritbot added a comment.Jun 19 2019, 6:34 PM

Change 517860 merged by jenkins-bot:
[integration/config@master] fresnel: Restore comment about firefox-esr

https://gerrit.wikimedia.org/r/517860

hashar closed this task as Resolved.Jun 19 2019, 6:35 PM
hashar assigned this task to Krinkle.

@Krinkle perfect thank you for the detailed explanation above. So Firefox is merely installed to ship extra dependencies, and probably we could instead just install the few that are missing. It just happens that the firefox-esr package depends on new enough libs that match the latest Chromium. So it is more or less a coincidence that it is working.

Regardless, yes that is small enough and the inline comment in the Dockerfile.template now properly explains why we got firefox installed. Thought for later: we could shrink the image by uninstalling chromium/firefox-esr since we only need their dependencies, but that is not much important.

Thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits