Page MenuHomePhabricator

All quibble-fresnel jobs crash before starting: "Could not find browser"
Closed, ResolvedPublic

Description

For example:

02:23:33 Error: Could not find browser revision 756035. Run "npm install" or "yarn install" to download a browser binary.
02:23:33     at ChromeLauncher.launch (/opt/fresnel/node_modules/fresnel/node_modules/puppeteer/lib/Launcher.js:81:23)

Build kept:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Nevermind, the JJB patch there is not yet applied to the live Jenkins job.

  • Last successful build: 45642 at Oct 6, 2021, 6:19 PM
  • First crash, which is the next build: 45643 at Oct 6, 2021, 7:49 PM

Screenshot 2021-10-07 at 05.33.59.png (1×720 px, 100 KB)

The first crash has a helpful config icon indicating a config change, with the following diff:

- 'docker-registry.wikimedia.org/releng/quibble-fresnel:1.0.1' --packages-source vendor --db mysql --db-dir /workspace/db --skip-deps --commands "mediawiki-fresnel-patch"
+ 'docker-registry.wikimedia.org/releng/quibble-fresnel:1.0.1-s1' --packages-source vendor --db mysql --db-dir /workspace/db --skip-deps --commands "mediawiki-fresnel-patch"

which corresponds with T279857

which has as the only nominal change:

-  https://getcomposer.org/download/1.10.22/composer.phar
+ https://getcomposer.org/download/2.1.8/composer.phar

which seems like it shouldn't affect whether puppeteer is still able to find the downloaded copy of Chromium.

Mentioned in SAL (#wikimedia-releng) [2021-10-07T04:46:25Z] <Krinkle> Temporarily change mediawiki-fresnel-patch-docker job change back to quibble-fresnel:1.0.1 ref T292684

Krinkle triaged this task as Medium priority.EditedThu, Oct 7, 4:48 AM

That fixed the issue for now. Lowering priority for now. This can investigated now without the on-going disruption. I might get to this in a few days, but anyone else feel free to give it a go as well.

Note: To my knowledge, the Fresnel jobs do not invoke composer so this does not actually require anything to remain compatible with composer 1.x.

A good first start might be to try to first confrim locally when fetching the old and new docker images from the registry whether the Chromium binary is indeed present before and now missing, and then to locally delete these images versions and build them anew with docker-pgk and see if it's deterministic or not.

@Krinkle did you change this in integration/config? If so, I'm not seeing it. I'd like to roll out this patch https://gerrit.wikimedia.org/r/c/integration/config/+/726880, I'll update that patch to keep quibble-fresnel at 1.0.1

I did it locally on Jenkins, I did not commit it.

Krinkle lowered the priority of this task from Medium to Low.
Krinkle edited projects, added Performance-Team (Radar); removed Performance-Team.

OK. Assigning over to you tentatively to investigate why custom commands are seemingly no longer executed in the new version (or executed differently?), but lowering priority since, for now, the old version will continue to work.

If you suspect the Fresnel job is doing something unsupported, or would like additional info, or otherwise think it's not likely to be worked on soon, then let us know, so that we can push it up the chain and have it settle elsewhere (or for me to try and get familiar with Quibble myself).

OK. Assigning over to you tentatively to investigate why custom commands are seemingly no longer executed in a new version, but lowering priority since, for now, the old version will continue to work.

I will try to find time to look at it but cc'ing @hashar in case this is something you're able to look at.

I have downloaded the build output from good and bad builds:

I have manually edited the few varying bits we don't care about such as the randomly generated /tmp/jenkinsXXXX.sh files, build numbers or timing. My finding:

ItemGoodBad
Imagereleng/quibble-fresnel:1.0.1releng/quibble-fresnel:1.0.1-s1
Composer1.10.222.1.8
NodeJS10.24.010.24.0
Npm6.14.57.5.2

Fresnel is installed in the image using npm install fresnel@1.0.0. I went with a dumb attempt at reproducing the issue running npm install fresnel@1.0.0 with npm 6.14.5 and 7.5.2 (in different directories. With npm 6.14.5, that triggers a download of Chromium r756035, with 7.5.2 it downloads nothing.

So I am guessing the issue is related to the npm upgrade.

Note: if I install fresnel with npm 6.14.5 then install it from a different directoy with 7.5.2, I will have chromium available. I am assuming chromium is stored in the cache when installing from 6.14.5 and npm 7.5.2 somehow manage to retrieve it from the cache.

Without cache, npm 7.5.2 does not download Chromium and that is all I got for tonight.

My system has Chromium 90.0.4430.212 from Debian Bullseye.

quibble-buster, which quibble-fresnel inherits from, installs its own copy of npm from Git which hasn't changed in a long time, but does so without pinning the tag, branch, or commit hash. I'm guessing it upgraded implicitly as part of rebuiding the parent layers at some random point, but for node10 we should stick to npm6, so that could be a small patch version to quibble-buster and confirm that that gets us back on track.

And then after that, or instead, perhaps we can try to upgrade from node10 to node12 in the quibble-buster image, which would be good to do regardless, possibly using the same install method as the standalone node12 image. That'll affect the qunit and selenium tests as well, which I guess also still use node10 today?

Change 730951 had a related patch set uploaded (by Krinkle; author: Krinkle):

[integration/config@master] jjb: Update fresnel job to quibble-fresnel:1.1.1-s1

https://gerrit.wikimedia.org/r/730951

Change 730951 merged by jenkins-bot:

[integration/config@master] jjb: Update fresnel job to quibble-fresnel:1.1.1-s1

https://gerrit.wikimedia.org/r/730951

Congratulations Timo, and thank you for fixing it.