For example:
02:23:33 Error: Could not find browser revision 756035. Run "npm install" or "yarn install" to download a browser binary. 02:23:33 at ChromeLauncher.launch (/opt/fresnel/node_modules/fresnel/node_modules/puppeteer/lib/Launcher.js:81:23)
Build kept:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| jjb: Update fresnel job to quibble-fresnel:1.1.1-s1 | integration/config | master | +1 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | kostajh | T292684 All quibble-fresnel jobs crash before starting: "Could not find browser" | |||
| Resolved | Krinkle | T293150 Update Fresnel to Puppeteer 7 (Chromium 90) |
The only recent change I could find to the job is "Quibble 1.1.0" (T291549)
The first crash has a helpful config icon indicating a config change, with the following diff:
- 'docker-registry.wikimedia.org/releng/quibble-fresnel:1.0.1' --packages-source vendor --db mysql --db-dir /workspace/db --skip-deps --commands "mediawiki-fresnel-patch" + 'docker-registry.wikimedia.org/releng/quibble-fresnel:1.0.1-s1' --packages-source vendor --db mysql --db-dir /workspace/db --skip-deps --commands "mediawiki-fresnel-patch"
which corresponds with T279857
which has as the only nominal change:
- https://getcomposer.org/download/1.10.22/composer.phar + https://getcomposer.org/download/2.1.8/composer.phar
which seems like it shouldn't affect whether puppeteer is still able to find the downloaded copy of Chromium.
Mentioned in SAL (#wikimedia-releng) [2021-10-07T04:46:25Z] <Krinkle> Temporarily change mediawiki-fresnel-patch-docker job change back to quibble-fresnel:1.0.1 ref T292684
That fixed the issue for now. Lowering priority for now. This can investigated now without the on-going disruption. I might get to this in a few days, but anyone else feel free to give it a go as well.
Note: To my knowledge, the Fresnel jobs do not invoke composer so this does not actually require anything to remain compatible with composer 1.x.
A good first start might be to try to first confrim locally when fetching the old and new docker images from the registry whether the Chromium binary is indeed present before and now missing, and then to locally delete these images versions and build them anew with docker-pgk and see if it's deterministic or not.
@Krinkle did you change this in integration/config? If so, I'm not seeing it. I'd like to roll out this patch https://gerrit.wikimedia.org/r/c/integration/config/+/726880, I'll update that patch to keep quibble-fresnel at 1.0.1
OK. Assigning over to you tentatively to investigate why custom commands are seemingly no longer executed in the new version (or executed differently?), but lowering priority since, for now, the old version will continue to work.
If you suspect the Fresnel job is doing something unsupported, or would like additional info, or otherwise think it's not likely to be worked on soon, then let us know, so that we can push it up the chain and have it settle elsewhere (or for me to try and get familiar with Quibble myself).
I will try to find time to look at it but cc'ing @hashar in case this is something you're able to look at.
I have downloaded the build output from good and bad builds:
I have manually edited the few varying bits we don't care about such as the randomly generated /tmp/jenkinsXXXX.sh files, build numbers or timing. My finding:
| Item | Good | Bad |
|---|---|---|
| Image | releng/quibble-fresnel:1.0.1 | releng/quibble-fresnel:1.0.1-s1 |
| Composer | 1.10.22 | 2.1.8 |
| NodeJS | 10.24.0 | 10.24.0 |
| Npm | 6.14.5 | 7.5.2 |
Fresnel is installed in the image using npm install fresnel@1.0.0. I went with a dumb attempt at reproducing the issue running npm install fresnel@1.0.0 with npm 6.14.5 and 7.5.2 (in different directories. With npm 6.14.5, that triggers a download of Chromium r756035, with 7.5.2 it downloads nothing.
So I am guessing the issue is related to the npm upgrade.
Note: if I install fresnel with npm 6.14.5 then install it from a different directoy with 7.5.2, I will have chromium available. I am assuming chromium is stored in the cache when installing from 6.14.5 and npm 7.5.2 somehow manage to retrieve it from the cache.
Without cache, npm 7.5.2 does not download Chromium and that is all I got for tonight.
My system has Chromium 90.0.4430.212 from Debian Bullseye.
quibble-buster, which quibble-fresnel inherits from, installs its own copy of npm from Git which hasn't changed in a long time, but does so without pinning the tag, branch, or commit hash. I'm guessing it upgraded implicitly as part of rebuiding the parent layers at some random point, but for node10 we should stick to npm6, so that could be a small patch version to quibble-buster and confirm that that gets us back on track.
And then after that, or instead, perhaps we can try to upgrade from node10 to node12 in the quibble-buster image, which would be good to do regardless, possibly using the same install method as the standalone node12 image. That'll affect the qunit and selenium tests as well, which I guess also still use node10 today?
Change 730951 had a related patch set uploaded (by Krinkle; author: Krinkle):
[integration/config@master] jjb: Update fresnel job to quibble-fresnel:1.1.1-s1
Change 730951 merged by jenkins-bot:
[integration/config@master] jjb: Update fresnel job to quibble-fresnel:1.1.1-s1