I've seen this a few times before. It's a race condition - as the video shows the form doesn't even get filled out:
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/58637/artifact/log/Talk-Add-discussion-for-talk-page-possible-as-logged-in-user.png

I'm not sure what the underlying cause is and how to deal with this edge case.

This time on a UniversalLanguageSelector patch

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/UniversalLanguageSelector/+/625664/

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/59175/console

I will need some help from @zeljkofilipin - looking at that test run the test is working it's just finishing too early before the page can reload.

Just to make sure I've understood the problem: MinervaNeue Selenium test fails sometimes while running in CI for other repositories? (This task mentions mediawiki/vendor and UniversalLanguageSelector.)

If that is the case, as the first step I would recommend disabling the test, because it's obviously flaky (replace it with it.skip). That will make sure the test doesn't cause trouble while it's being fixed.

As the second step, I would recommend running the test 10-100 times in a row to see if the failure can be reproduced locally. If it can be reproduced, that's a good step towards fixing it. It should be as easy as:

for i in {1..10}; do npm run selenium-test -- --spec tests/selenium/specs/talk.js --mochaOpts.grep newly; done

If it can't be reproduced, as the third step I would recommend running the entire test suite 10-100 times in a row. Maybe the test fails only when running after other tests. (That should not happen, but sometimes does.) It should be as easy as:

for i in {1..10}; do npm run selenium-test; done

Let me know if you need help with any of the suggested steps.

Also affects Wikibase (change 626874, quibble-vendor-mysql-php72-selenium-docker #3823).

I ran the entire test suite 5 times on my machine, targeting mediawiki-docker:

for i in {1..5}; do npm run selenium-test; done
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:33 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:33 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:43 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:28 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:39 
...

For the entire terminal output see P12609.

I ran the failing test 10 times on my machine, targeting mediawiki-docker:

for i in {1..10}; do npm run selenium-test -- --spec tests/selenium/specs/talk.js --mochaOpts.grep newly; done
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:39 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:36 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:35 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:39 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:37 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:40 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...

For the entire terminal output see P12610.

Change 628947 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Skip test

https://gerrit.wikimedia.org/r/628947

In T262617#6463168, @Jdlrobson wrote:

I will need some help from @zeljkofilipin - looking at that test run the test is working it's just finishing too early before the page can reload.

I'm not sure if you were expecting more help from me than just debugging advice. I did take a look at MinervaNeue Selenium tests and I can't really help much because it's not using the usual Selenium stack. I'm not familiar with your stack (Cucumber instead of Mocha, for example) so it would take me a long time to debug the problem. The only help I can offer is in refactoring the code to use our usual stack. I've started working on that with @Edtadros in T245250: Review MinervaNeue Selenium tests.

I think we can just disable this test for now. Was hoping it was just a blip. Is there a way to disable a test so it is only disabled on CI?

In T262617#6487558, @Jdlrobson wrote:

Is there a way to disable a test so it is only disabled on CI?

All CI is doing is calling the selenium-test npm script.

"selenium-test": "npm -s run test:size && wdio tests/selenium/wdio.conf.js"

You can change the script to only run a subset of tests using --mochaOpts.grep. We do that in core when running daily tests:

"selenium-daily": "npm run selenium-test -- --mochaOpts.grep @daily",

If you change talk.js from

it( 'A newly created topic appears in the list of topics', () => {

to

it( 'A newly created topic appears in the list of topics @skip', () => {

There's probably a way to exclude just this one spec with @skip. --mochaOpts.grep will include tests with the string. I'm not sure how to exclude tests, but there's probably a way. Let me know if that's what you had in mind, I can take a closer look.

There are more options at https://webdriver.io/docs/organizingsuites.html

Just a note that this is still occuring as of today https://gerrit.wikimedia.org/r/c/mediawiki/core/+/629514

Change 628947 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Skip Minerva selenium test for creating talk topic

https://gerrit.wikimedia.org/r/628947