Cindy-the-browser-test-bot has been disabled for a week now as it started regularly failing some of the tests, unrelated to the actual patch under test. Figure out why the tests are failing and make them start working.
AC:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| integration: Always set baseUrl in wdio.conf | mediawiki/extensions/CirrusSearch | master | +2 -0 |
Monitoring the test run through the chrome inspector (port forward 9222 into the cloud instance, and then again into the vagrant instance, then visit chrome://inspect in a chromium based browser) shows that the strings we are telling the browser to write into the <input> box isn't actually making it there. Most of the time it makes it there, but sometimes it loses a letter.
Random searching for things related to webdriverio, chrome-driver/chromium-driver and incomplete inputs suggests this happens when javascript handlers attach to an input and run "too slowly". Related tickets on the webdriverio library we use says these bugs are part of the driver being used (chromium-driver, v73) and can't be fixed on the client side (inside the test suite).
Running a git bisect against the Vector repo turns up a patch where we started regularly failing: Use Codex and Vue 3 instead of WVUI and Vue 2. This is a fairly significant rework of the search input box, we aren't going to be able to do anything about this specific patch.
mediawiki-vagrant only runs in a debian 9 container, we are fairly stuck to the last version of chromium-driver that has available debian 9 packages. One alternate option is to run chromedriver on the host (debian 10, chromium v90) and port forward that into the vagrant instance.
Change 819152 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[mediawiki/extensions/CirrusSearch@master] integration: Always set baseUrl in wdio.conf
Change 819152 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] integration: Always set baseUrl in wdio.conf
Between vacations and other activities hadn't had a chance to look into this since aug 1st. Upon returning from vacation cirrus-integ.eqiad.wmflabs had filled it's disk and managed to corrupt various files inside the vagrant instance. Getting it going again meant destroying the vagrant instance and recreating it. That is now complete and the bot is back up and running again.
There are two failures while testing the es710 branch, will look into them closer this afternoon with the chrome debugger and see what is going on, likely we can get these passing as well and declare success soon.
Error in "Smoke test: Search suggestions: Then suggestions should appear" [2022-08-18 18:33:08+00:00] waitUntil condition timed out after 20000ms
[2022-08-18 18:33:11+00:00] [0-35] Error in "Smoke test: Search with accent yields result page with accent: Then I am on a page titled África" [2022-08-18 18:33:11+00:00] I am on África: expected 'Search results' to equal 'África'
Above test failure was unrelated to our test runner or elastic 7.10, rather a patch merged to Vector earlier today doesn't seem to work as expected. I've notified the patch author and gotten a succesfull run out of the integration suite by testing against master~1 of the Vector skin.
For followup on what exactly fixed the runner, it was mostly down to the comment from earlier
One alternate option is to run chromedriver on the host (debian 10, chromium v90) and port forward that into the vagrant instance.
chromedriver now runs on the integration testing host rather than inside the mwvagrant instance and is port forwarded into the instance. This allowed us to get a newer (but not that new, only chrome v90. Chrome v104 is latest) instance of chrome that no longer appears to have the problem with dropping some key presses while adding to the input boxes.