Moving to External because this isn't within the AW team's control. @vaughnwalters , is that reasonable?

@cmassaro yeah, I think so.

@Waddie96: You have moved this from External where this task was triaged (see above) to current planned work this Quarter. Can you explain why?

Noting here that Catalyst folks created this CI job, but are not planning to take action on this task since we're unsure how that CI variable controls browser version. @Peter are you planning on looking at this?

It works like this: With later version of webdriver.io it internally downloads the browser if not the browser binary is pointed out in the configuration. By default it download the latest stable version and then try to download the same driver version. This could potentially be good when run test locally but in CI it's broken. The way we handled it is that we look for if a env variable CI is set. If it is, we configure webdriver to use the browser and driver that already exists in the Docker container. I guess then that the CI variable only exists for some job. Either we add it or we can switch on another environment variable, which one should be safe to use (=only set in CI), do you know @thcipriani ?

In T407763#11394902, @Peter wrote:

It works like this: With later version of webdriver.io it internally downloads the browser if not the browser binary is pointed out in the configuration. By default it download the latest stable version and then try to download the same driver version. This could potentially be good when run test locally but in CI it's broken. The way we handled it is that we look for if a env variable CI is set. If it is, we configure webdriver to use the browser and driver that already exists in the Docker container.

It seems safer to set this variable by default in the Docker image, and over-ride it to blank if manual testers want to experiment?

It seems safer to set this variable by default in the Docker image, and over-ride it to blank if manual testers want to experiment?

As it is now manual tests will use webdriverIO default, meaning it will download latest stable browser/driver version. But as you said, we could export the paths in Docker, let me check if webdriverIO have any env variables they use by default.

Me and @zeljkofilipin debugged this today and it looks like this (and other daily jobs) is based on another Docker container than the core jobs, and that container comes with Chrome/Chromedriver 140. I wonder if there's a reason for that (built upon different containers)?

However, we would love to upgrade to 140 (from 120) that would help us to enable things like T410246 - we could run tests as recommended but the webdriverIO team.

In T407763#11416052, @Peter wrote:

Me and @zeljkofilipin debugged this today and it looks like this (and other daily jobs) is based on another Docker container than the core jobs, and that container comes with Chrome/Chromedriver 140. I wonder if there's a reason for that (built upon different containers)?

If you mean the Quibble jobs, yes, those are based on the Quibble images (currently quibble-bullseye-php83, from integration/config.git), whereas Catalyst is based on the developer images (from releng/dev-images).

However, we would love to upgrade to 140 (from 120) that would help us to enable things like T410246 - we could run tests as recommended but the webdriverIO team.

As of Friday we switched from bookworm-php81-fpm to bookworm-php83-fpm, which incidentally bumped Chrome to 140, so that part of this is "fixed".

If you want the Catalyst jobs to use the Quibble CI images rather than the more prod-like dev images, as implied, this task should probably be kept open for that.

In T407763#11419717, @Jdforrester-WMF wrote:

As of Friday we switched from bookworm-php81-fpm to bookworm-php83-fpm, which incidentally bumped Chrome to 140, so that part of this is "fixed".

Is this for quibble jobs? I took a look at a recent core patch (1205686) and both quibble-with-gated-extensions-selenium-php83 (840) and quibble-for-mediawiki-core-browser-tests-only-vendor-mysql-php83 (304) run chrome 120.

Example:

...
[chrome 120.0.6099.224 linux #0-0] Running: chrome (v120.0.6099.224) on linux
...

In T407763#11423865, @zeljkofilipin wrote:

In T407763#11419717, @Jdforrester-WMF wrote:

As of Friday we switched from bookworm-php81-fpm to bookworm-php83-fpm, which incidentally bumped Chrome to 140, so that part of this is "fixed".

Is this for quibble jobs? I took a look at a recent core patch (1205686) and both quibble-with-gated-extensions-selenium-php83 (840)and quibble-for-mediawiki-core-browser-tests-only-vendor-mysql-php83 (304) run chrome 120.

Example:

...
[chrome 120.0.6099.224 linux #0-0] Running: chrome (v120.0.6099.224) on linux
...

Hmm, you're right; I should poke that to update CI to a more modern browser.

What code changes would need to be made to make this happen? That will help determine what team needs to make the changes:

If it's in core catalyst code or some shared WDIO setting or something, that might be Test Platform? But if it's in Abstract Wikipedia test code, QS-Test-Automation can handle it or Abstract Wikipedia can do it themselves with us helping troubleshoot.

Re-titling this to be more specific about the "problem". I don't think the right fix here is to regress the browser in Catalyst, but to upgrade the browser in legacy CI to one that our users actually use.

To be clear, I agree. I'm just trying to figure out who needs to do that work based on where the code lives that needs to be updated.

In T407763#11563532, @SLong-WMF wrote:

To be clear, I agree. I'm just trying to figure out who needs to do that work based on where the code lives that needs to be updated.

The legacy CI run off our Quibble images (currently, this one), which are maintained in the integration/config.git repo mostly by @hashar and me. The reason they're old is because they're directly based off Debian Bullseye, to try to match the OS version in production, but we could ignore this concern and jump ahead. Moving to Bookworm is T362705; moving to Trixie is T397050. I think it'd probably be fine to do this for selenium tests (but not the rest).

In T407763#11563551, @Jdforrester-WMF wrote:

In T407763#11563532, @SLong-WMF wrote:

To be clear, I agree. I'm just trying to figure out who needs to do that work based on where the code lives that needs to be updated.

The legacy CI run off our Quibble images (currently, this one), which are maintained in the integration/config.git repo mostly by @hashar and me. The reason they're old is because they're directly based off Debian Bullseye, to try to match the OS version in production, but we could ignore this concern and jump ahead. Moving to Bookworm is T362705; moving to Trixie is T397050. I think it'd probably be fine to do this for selenium tests (but not the rest).

Yes, I also think should upgrade for the selenium tests. IMO if selenium tests break we will fix them but we should be testing against versions that real users will actually be using.

Based on refinement this morning:

This feels like it should involve Release engineering. I'll talk with that team and figure out how to proceed.

Ideally, we simply need to align all of the versions of Chromium into one version and make sure we're not inadvertently blowing up the release pipeline.

@SLong-WMF what was the outcome of that conversation?

I chatted with Tyler, this is a QS + RelEng + TP collaboration. QS will own and drive this forward.

Note: as of just now (via T362705) main CI jobs are using Chromium 149 (up from 120).

149 is not the same as 140 used in Catalyst's CI, but it's different.