Page MenuHomePhabricator

Minerva Selenium test for talk page creation was disabled (Can't call getText on element with selector ".section-heading" because element wasn't found)
Open, LowPublic

Description

Completely unrelated change
https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/621952/
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/58637/console

22:36:10 [Chrome 73.0.3683.75 linux #0-7] Spec: /workspace/src/skins/MinervaNeue/tests/selenium/specs/talk.js
22:36:10 [Chrome 73.0.3683.75 linux #0-7] Running: Chrome (v73.0.3683.75) on linux
22:36:10 [Chrome 73.0.3683.75 linux #0-7] Session ID: 11f27abe-7a31-447f-8bc3-922c99100e6f
22:36:10 [Chrome 73.0.3683.75 linux #0-7]
22:36:10 [Chrome 73.0.3683.75 linux #0-7] Talk
22:36:10 [Chrome 73.0.3683.75 linux #0-7] ✓ Talk button not visible as logged out user
22:36:10 [Chrome 73.0.3683.75 linux #0-7] ✓ Talk tab visible as logged in user
22:36:10 [Chrome 73.0.3683.75 linux #0-7] ✓ Talk on a page that doesn't exist (bug 64268)
22:36:10 [Chrome 73.0.3683.75 linux #0-7] ✓ Add discussion button shows on talk pages for logged in users
22:36:10 [Chrome 73.0.3683.75 linux #0-7] ✓ Add discussion for talk page possible as logged in user
22:36:10 [Chrome 73.0.3683.75 linux #0-7] ✖ A newly created topic appears in the list of topics
22:36:10 [Chrome 73.0.3683.75 linux #0-7]
22:36:10 [Chrome 73.0.3683.75 linux #0-7] 5 passing (36.4s)
22:36:10 [Chrome 73.0.3683.75 linux #0-7] 1 failing
22:36:10 [Chrome 73.0.3683.75 linux #0-7]
22:36:10 [Chrome 73.0.3683.75 linux #0-7] 1) Talk A newly created topic appears in the list of topics
22:36:10 [Chrome 73.0.3683.75 linux #0-7] Can't call getText on element with selector ".section-heading" because element wasn't found
22:36:10 [Chrome 73.0.3683.75 linux #0-7] Error: Can't call getText on element with selector ".section-heading" because element wasn't found
22:36:10 [Chrome 73.0.3683.75 linux #0-7] at iShouldSeeTheTopicInTheListOfTopics (/workspace/src/skins/MinervaNeue/tests/selenium/features/step_definitions/talk_steps.js:52:37)
22:36:10 [Chrome 73.0.3683.75 linux #0-7] at Context.it (/workspace/src/skins/MinervaNeue/tests/selenium/specs/talk.js:75:3)

Event Timeline

Jdlrobson added a subscriber: Jdlrobson.

I've seen this a few times before. It's a race condition - as the video shows the form doesn't even get filled out:
https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/58637/artifact/log/Talk-Add-discussion-for-talk-page-possible-as-logged-in-user.png

I'm not sure what the underlying cause is and how to deal with this edge case.

I will need some help from @zeljkofilipin - looking at that test run the test is working it's just finishing too early before the page can reload.

Just to make sure I've understood the problem: MinervaNeue Selenium test fails sometimes while running in CI for other repositories? (This task mentions mediawiki/vendor and UniversalLanguageSelector.)

If that is the case, as the first step I would recommend disabling the test, because it's obviously flaky (replace it with it.skip). That will make sure the test doesn't cause trouble while it's being fixed.

As the second step, I would recommend running the test 10-100 times in a row to see if the failure can be reproduced locally. If it can be reproduced, that's a good step towards fixing it. It should be as easy as:

for i in {1..10}; do npm run selenium-test -- --spec tests/selenium/specs/talk.js --mochaOpts.grep newly; done

If it can't be reproduced, as the third step I would recommend running the entire test suite 10-100 times in a row. Maybe the test fails only when running after other tests. (That should not happen, but sometimes does.) It should be as easy as:

for i in {1..10}; do npm run selenium-test; done

Let me know if you need help with any of the suggested steps.

I ran the entire test suite 5 times on my machine, targeting mediawiki-docker:

for i in {1..5}; do npm run selenium-test; done
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:33 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:33 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:43 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:28 
...
Spec Files:	 10 passed, 10 total (100% completed) in 00:04:39 
...

For the entire terminal output see P12609.

I ran the failing test 10 times on my machine, targeting mediawiki-docker:

for i in {1..10}; do npm run selenium-test -- --spec tests/selenium/specs/talk.js --mochaOpts.grep newly; done
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:39 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:36 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:35 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:39 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:37 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:40 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...
Spec Files:	 1 passed, 1 total (100% completed) in 00:00:38 
...

For the entire terminal output see P12610.

Change 628947 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Skip test

https://gerrit.wikimedia.org/r/628947

I will need some help from @zeljkofilipin - looking at that test run the test is working it's just finishing too early before the page can reload.

I'm not sure if you were expecting more help from me than just debugging advice. I did take a look at MinervaNeue Selenium tests and I can't really help much because it's not using the usual Selenium stack. I'm not familiar with your stack (Cucumber instead of Mocha, for example) so it would take me a long time to debug the problem. The only help I can offer is in refactoring the code to use our usual stack. I've started working on that with @Edtadros in T245250: Review MinervaNeue Selenium tests.

I think we can just disable this test for now. Was hoping it was just a blip. Is there a way to disable a test so it is only disabled on CI?

Is there a way to disable a test so it is only disabled on CI?

All CI is doing is calling the selenium-test npm script.

"selenium-test": "npm -s run test:size && wdio tests/selenium/wdio.conf.js"

You can change the script to only run a subset of tests using --mochaOpts.grep. We do that in core when running daily tests:

"selenium-daily": "npm run selenium-test -- --mochaOpts.grep @daily",

If you change talk.js from

it( 'A newly created topic appears in the list of topics', () => {

to

it( 'A newly created topic appears in the list of topics @skip', () => {

There's probably a way to exclude just this one spec with @skip. --mochaOpts.grep will include tests with the string. I'm not sure how to exclude tests, but there's probably a way. Let me know if that's what you had in mind, I can take a closer look.

There are more options at https://webdriver.io/docs/organizingsuites.html

Change 628947 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Skip Minerva selenium test for creating talk topic

https://gerrit.wikimedia.org/r/628947

Jdlrobson renamed this task from Can't call getText on element with selector ".section-heading" because element wasn't found to Minerva Selenium test for talk page creation was disabled (Can't call getText on element with selector ".section-heading" because element wasn't found).Sep 24 2020, 3:00 PM
Jdlrobson triaged this task as Low priority.
Jdlrobson removed a project: Patch-For-Review.
Jdub252 renamed this task from Minerva Selenium test for talk page creation was disabled (Can't call getText on element with selector ".section-heading" because element wasn't found) to Mine.Sep 28 2020, 12:15 AM
Jdub252 closed this task as Resolved.
Jdub252 updated the task description. (Show Details)
Jdub252 removed subscribers: Ammarpad, Edtadros, Michael and 6 others.
Meno25 renamed this task from Mine to Minerva Selenium test for talk page creation was disabled (Can't call getText on element with selector ".section-heading" because element wasn't found).Sep 28 2020, 3:36 AM
Meno25 reopened this task as Open.
Meno25 updated the task description. (Show Details)
Meno25 added subscribers: Ammarpad, Edtadros, Michael and 6 others.