Page MenuHomePhabricator

Improve rollback confirmation selenium tests
Open, Needs TriagePublic1 Story Points

Description

Right now, the tests related to the rollback confirmation prompt (specifically tests/selenium/specs/rollback.js) use browser.pause() to solve issues with delays in JavaScript loading and elements appearing after animations have been executed.

In order to solve the issues with the delayed loading of JavaScript modules, we should implement something similar to what has been done in the Cite extension to ensure that mediawiki.api is always loaded before we try to make API calls:

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/502212/

Additionally, instead of using pause to wait for the buttons to become clickable, we should use waitForVisible() to ensure the elements are there without relying on arbitrary time constraints.

Event Timeline

Tim_WMDE created this task.Apr 9 2019, 9:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 9 2019, 9:15 AM

Change 503951 had a related patch set uploaded (by Tim Eulitz; owner: Tim Eulitz):
[mediawiki/core@master] Replace pauses in rollback tests w/ dynamic waits

https://gerrit.wikimedia.org/r/503951

Tim_WMDE set the point value for this task to 2.Apr 15 2019, 10:25 AM
Tim_WMDE moved this task from Todo to Review on the WMDE-FUN-Sprint-2019-04-01 board.
kai.nissen changed the point value for this task from 2 to 0.
kai.nissen moved this task from Todo to Review on the WMDE-FUN-Sprint-2019-04-15 board.
Tim_WMDE changed the point value for this task from 0 to 1.Apr 15 2019, 3:25 PM

Change 506623 had a related patch set uploaded (by Tonina Zhelyazkova; owner: Tonina Zhelyazkova):
[mediawiki/core@master] Improve rollback tests setup by extracting repeating logic to HistoryPage object

https://gerrit.wikimedia.org/r/506623

Change 503951 merged by jenkins-bot:
[mediawiki/core@master] Replace pauses in rollback tests w/ dynamic waits

https://gerrit.wikimedia.org/r/503951

Change 508090 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Remove LoginPage.waitForScriptsToBeReady() method

https://gerrit.wikimedia.org/r/508090

Krinkle added a subscriber: Krinkle.May 4 2019, 5:08 PM

Please convert to mw.loader.using() so that it doesn't become someone else's problem when the tests fail again if mediawiki.api isn't used by the log in page for some reason. This is a test for the rollback feature, not for the login page. And neither of these enqueues mediawiki.api. It is needed for an inline snippet, so should be ensured by that inline snippet.

Please convert to mw.loader.using() so that it doesn't become someone else's problem when the tests fail again if mediawiki.api isn't used by the log in page for some reason. This is a test for the rollback feature, not for the login page. And neither of these enqueues mediawiki.api. It is needed for an inline snippet, so should be ensured by that inline snippet.

Hey, you're right this is something we did not consider yet ( and I guess it's also where we were talking about different things but thought we meant the same ^^' ). So I had a quick talk with the team. In the end we need an approach that does a bit both.

  • We have to use mw.loader.using() to make sure the modules are not only there but also will be loaded in the first place.
  • We also have to use some wait step to make sure that the MW JS core lib is loaded so that even can use mw.loader

We will change the waitForScripts method so that it does both. And this method can then live as a general "wait for and load a module" method in the Utils.

Thanks again for all the time spend in reviewing this. I hope we found a good ending now.

Change 506623 merged by jenkins-bot:
[mediawiki/core@master] Improve rollback tests setup by extracting repeating logic to HistoryPage object

https://gerrit.wikimedia.org/r/506623

Change 508714 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/core@master] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508714

Change 508714 merged by jenkins-bot:
[mediawiki/core@master] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508714

Change 508849 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/core@wmf/1.34.0-wmf.4] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508849

Change 508850 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/core@wmf/1.34.0-wmf.3] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508850

Change 508849 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.4] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508849

Change 508850 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.3] Temporarily skip rollback test

https://gerrit.wikimedia.org/r/508850

Change 508938 had a related patch set uploaded (by Krinkle; owner: Jdlrobson):
[mediawiki/core@master] selenium: Asynchronous methods should be returned to browser.call()

https://gerrit.wikimedia.org/r/508938

Change 508090 abandoned by Krinkle:
selenium: Remove LoginPage.waitForScriptsToBeReady() method

Reason:
Superseded by rMW3990e0bbc709: Improve rollback tests setup by extracting repeating logic to HistoryPage object, which already fixed the issue of the extra method that was added in the public wdio-mediawiki package.

https://gerrit.wikimedia.org/r/508090

Change 508938 merged by jenkins-bot:
[mediawiki/core@master] selenium: Asynchronous methods should be returned to browser.call()

https://gerrit.wikimedia.org/r/508938

Got another instance:

11:09:48 1 failing
11:09:48 
11:09:48 1) Rollback with confirmation should offer a way to cancel rollbacks:
11:09:48 element (".mw-rollback-link .jquery-confirmable-button-no") still not visible after 5000ms
11:09:48 running chrome
11:09:48 Error: element (".mw-rollback-link .jquery-confirmable-button-no") still not visible after 5000ms
11:09:48     at Context.<anonymous> (tests/selenium/specs/rollback.js:47:37)
11:09:48     at Promise.F (node_modules/core-js/library/modules/_export.js:36:28)
11:09:48     at elements(".mw-rollback-link .jquery-confirmable-button-no") - isVisible.js:54:17
11:09:48     at isVisible(".mw-rollback-link .jquery-confirmable-button-no") - waitForVisible.js:73:22

Patch: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/505282

And again at https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/516308/

[chrome #0-2]   Rollback with confirmation
[chrome #0-2]       ✓ should offer rollback options for admin users
[chrome #0-2]       - should offer a way to cancel rollbacks
[chrome #0-2]       - should perform rollbacks after confirming intention
[chrome #0-2]       1) should verify rollbacks via GET requests are confirmed on a follow-up page
[chrome #0-2]
[chrome #0-2]   Rollback without confirmation
[chrome #0-2]       ✓ should perform rollback via POST request without asking the user to confirm
[chrome #0-2]       ✓ should perform rollback via GET request without asking the user to confirm
[chrome #0-2]
[chrome #0-2]
[chrome #0-2] 3 passing (22s)
[chrome #0-2] 2 pending
[chrome #0-2] 1 failing
[chrome #0-2]
[chrome #0-2] 1) Rollback with confirmation should verify rollbacks via GET requests are confirmed on a follow-up page:
[chrome #0-2] Expected rollback page to appear.
[chrome #0-2] Error: Expected rollback page to appear.
[chrome #0-2]     at getText("#firstHeading") - at elements("#firstHeading") - getText.js:18:17
[chrome #0-2]

Change 516810 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Skip another flaky Rollback test

https://gerrit.wikimedia.org/r/516810

Change 516810 merged by jenkins-bot:
[mediawiki/core@master] selenium: Skip another flaky Rollback test

https://gerrit.wikimedia.org/r/516810

Krinkle added a comment.EditedJun 17 2019, 12:53 AM

I guess it's still not well-enough disabled. Still failing builds at gerrit 517348

 1) Rollback with confirmation should offer rollback options for admin users:
An element could not be located on the page using the given search parameters (".mw-rollback-link .jquery-confirmable-text").
running chrome
Error: An element could not be located on the page using the given search parameters (".mw-rollback-link .jquery-confirmable-text").
    at getText() - index.js:312:3

.. and again at gerrit 512067

1) Rollback without confirmation should perform rollback via GET request without asking the user to confirm:
Expected rollback page to appear.
running chrome
Error: Expected rollback page to appear.
    at elementIdText("0.007272742124479814-1") - getText.js:35:50
mmodell changed the subtype of this task from "Task" to "Production Error".Wed, Aug 28, 11:07 PM