@zeljkofilipin when you have some time, your help would be greatly appreciated :)

Apologies for being slow on this. Looks like this is caused by a dependency upgrade, 651862.

wdio-mediawiki 1.0.0 "requires webdriverio 5". CirrusSearch is at "webdriverio": "4.13.2".

Change 656180 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/extensions/CirrusSearch@master] build: Revert updating wdio-mediawiki to 1.0.0

https://gerrit.wikimedia.org/r/656180

Proving that the problem is solved by the above commit.

In one terminal tab/window, run Chromedriver.

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ chromedriver --url-base=wd/hub --port=4444
(...)

Run the rest of the commands in another terminal tab/window.

Export required environment variable.

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ export MW_SERVER=https://en.wikipedia.beta.wmflabs.org

Reproduce the error on master branch.

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ git checkout master
(...)

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ npm ci
(...)

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ npm run selenium-daily       
(...)
0 passing (1.30s)
3 failing
(...)
Cannot read property 'baseUrl' of undefined
(...)

Get the patch from Gerrit and run the tests.

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ git review -d 656180
(...)

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ npm ci
(...)

~/Documents/gerrit/mediawiki/core/extensions/CirrusSearch$ npm run selenium-daily       
(...)
3 passing (11.50s)

This problem is caused by LibUp automatically upgrading a dependency. The problem is that the tool updated MAJOR version of the dependency. It upgraded wdio-mediawiki 0.4.0 to 1.0.0. The CI didn't catch the introduced problem immediately, because CirrusSearch Selenium tests don't run in CI. Cindy-the-browser-test-bot was also happy with the change. The only thing that failed is a job that runs daily and targets the beta cluster. Because there's a delay between the merge and the job run (additionally, it was Christmas eve), nobody noticed the problem until 2021-01-04. It took me a while to catch up after vacation, but the above patch should solve the problem.

Change 656180 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] build: Revert updating wdio-mediawiki to 1.0.0

https://gerrit.wikimedia.org/r/656180

In T271102#6747887, @zeljkofilipin wrote:

This problem is caused by LibUp automatically upgrading a dependency. The problem is that the tool updated MAJOR version of the dependency. It upgraded wdio-mediawiki 0.4.0 to 1.0.0.

The tool was intentionally told to do this in https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/config/+/651861

The CI didn't catch the introduced problem immediately, because CirrusSearch Selenium tests don't run in CI.

That's a footgun we're trying to ban. Can we fix the repo?

Cindy-the-browser-test-bot was also happy with the change. The only thing that failed is a job that runs daily and targets the beta cluster. Because there's a delay between the merge and the job run (additionally, it was Christmas eve), nobody noticed the problem until 2021-01-04. It took me a while to catch up after vacation, but the above patch should solve the problem.

LibUp will blindly retry this tonight, BTW.

In T271102#6749334, @Jdforrester-WMF wrote:

In T271102#6747887, @zeljkofilipin wrote:

This problem is caused by LibUp automatically upgrading a dependency. The problem is that the tool updated MAJOR version of the dependency. It upgraded wdio-mediawiki 0.4.0 to 1.0.0.

The tool was intentionally told to do this in https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/config/+/651861

My bad, I probably should have looked into it more as a major version bump.

The CI didn't catch the introduced problem immediately, because CirrusSearch Selenium tests don't run in CI.

That's a footgun we're trying to ban. Can we fix the repo?

Yes please. LibUp works on the assumption that Jenkins actually tests all the dependencies before merging patches.

Cindy-the-browser-test-bot was also happy with the change. The only thing that failed is a job that runs daily and targets the beta cluster. Because there's a delay between the merge and the job run (additionally, it was Christmas eve), nobody noticed the problem until 2021-01-04. It took me a while to catch up after vacation, but the above patch should solve the problem.

LibUp will blindly retry this tonight, BTW.

I reverted the LibUp config change for now so it won't try to do it again.

In T271102#6749334, @Jdforrester-WMF wrote:

In T271102#6747887, @zeljkofilipin wrote:

This problem is caused by LibUp automatically upgrading a dependency. The problem is that the tool updated MAJOR version of the dependency. It upgraded wdio-mediawiki 0.4.0 to 1.0.0.

The tool was intentionally told to do this in https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/config/+/651861

The CI didn't catch the introduced problem immediately, because CirrusSearch Selenium tests don't run in CI.

That's a footgun we're trying to ban. Can we fix the repo?

Depends what fix the repo means? Our main problem in getting things moved over is that we need wikis in multiple languages as well as a commonswiki to test the functionality we deploy to prod. I can't say i've spent significant time on it, but i've taken a day before to try and make quibble do it but I couldn't really get a multi-wiki going. It's been awhile so maybe things are closer to what we need today?

I'm also not sure why cindy +1'd the patches, i remember at some point we reduced how often it was running npm because it was a noticable part of the runtime. Perhaps it simply continued using the existing vendored libraries.

Depends what fix the repo means? Our main problem in getting things moved over is that we need wikis in multiple languages as well as a commonswiki to test the functionality we deploy to prod. I can't say i've spent significant time on it, but i've taken a day before to try and make quibble do it but I couldn't really get a multi-wiki going. It's been awhile so maybe things are closer to what we need today?

I guess we would need a custom script job that is able to provision the multiple databases and craft a specific LocalSettings.php which is multiple versions aware. In the ideal world of magic, we would use operations-software/mediawiki-config to spin up a full test environment, but we are really really far from being able to achieve that kind of full scale testing.

I am open to have a dedicated job that is tweak to support that specific use case. It seems worthwhile to have. But maybe that is what Cindy the Browser bot is for anyway?

In T271102#6750736, @hashar wrote:

I am open to have a dedicated job that is tweak to support that specific use case. It seems worthwhile to have. But maybe that is what Cindy the Browser bot is for anyway?

Yes this is the role of this bot, it's based on vagrant as this is sole fully-featured dev environment we currently have for CirrusSearch. I would take this as an opportunity to move away from vagrant for CirrusSearch but not sure if we have the bandwidth to work on this anytime soon.

As to fixing the repo would upgrading "webdriverio" to 5 be sufficient as a short term fix?

In T271102#6750860, @dcausse wrote:

As to fixing the repo would upgrading "webdriverio" to 5 be sufficient as a short term fix?

@Vidhi-Mody has tried that in T257525: Upgrade WebdriverIO in the CirrusSearch repository but failed because she couldn't set up local development environment. Updating from webdriverio 4 to 5 will be some work. If that is done, updating from 5 to 6 (current version) should be almost trivial. (Speaking from experience with T247844: Upgrade WebdriverIO to v6 in all repositories.)

In T271102#6750860, @dcausse wrote:

As to fixing the repo would upgrading "webdriverio" to 5 be sufficient as a short term fix?

I think 655940 is already a short term fix:

In T271102#6749497, @Legoktm wrote:

I reverted the LibUp config change for now so it won't try to do it again.

Change 753436 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[labs/libraryupgrader/config@master] wdio-mediawiki: Bump to 1.2.0

https://gerrit.wikimedia.org/r/753436

Change 753436 merged by jenkins-bot:

[labs/libraryupgrader/config@master] wdio-mediawiki: Bump to 1.2.0

https://gerrit.wikimedia.org/r/753436