The selenium-daily-beta-Minerva job has been failing since the beginning of May. This job runs the full Minerva selenium test suite targeting the beta cluster on a daily basis. The test logs as well as build artifacts are available on Jenkins .
The first tests failure occurs on the "Wikitext Editor (Makes actual saves)" test suite. There are 3 tests in this file:
// @test2.m.wikipedia.org @login describe( 'Wikitext Editor (Makes actual saves)', () => { beforeEach( () => { iAmLoggedIntoTheMobileWebsite(); } ); // @editing it( 'It is possible to edit', () => { iGoToAPageThatHasLanguages(); iClickTheEditButton(); iSeeTheWikitextEditorOverlay(); iTypeIntoTheEditor( 'ABC GHI' ); iClickContinue(); iClickSubmit(); iDoNotSeeTheWikitextEditorOverlay(); iShouldSeeAToastNotification(); } ); // @editing @en.m.wikipedia.beta.wmflabs.org it( 'Redirects', () => { const title = 'Selenium wikitext editor test ' + Math.random(); pageExists( title ); iAmOnAPageThatDoesNotExist(); iClickTheEditButton(); iSeeTheWikitextEditorOverlay(); iClearTheEditor(); iTypeIntoTheEditor( `#REDIRECT [[${title}]]` ); iClickContinue(); iClickSubmit(); iSayOkayInTheConfirmDialog(); iDoNotSeeTheWikitextEditorOverlay(); theTextOfTheFirstHeadingShouldBe( title ); } ); // @editing @en.m.wikipedia.beta.wmflabs.org it( 'Broken redirects', () => { iAmOnAPageThatDoesNotExist(); iClickTheEditButton(); iSeeTheWikitextEditorOverlay(); iClearTheEditor(); iTypeIntoTheEditor( '#REDIRECT [[AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA]]' ); iClickContinue(); iClickSubmit(); iSayOkayInTheConfirmDialog(); iDoNotSeeTheWikitextEditorOverlay(); thereShouldBeARedLinkWithText( 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' ); } ); } );
See https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-daily-beta-Minerva/80/ as an example build that failed. The tests complain of:
- badtoken: Invalid CSRF token.
- Could not login: WrongToken
- element (".overlay .wikitext-editor") still not existing after 10000ms
Reproduction steps
To run the Selenium tests against the beta cluster on your local machine, ensure you have chromedriver installed, then npm install.
Start chromedriver in one tab:
$> chromedriver --url-base=/wd/hub --port=4444
and start the tests in another:
$> MEDIAWIKI_PASSWORD={from officewiki} \ MEDIAWIKI_USER="Selenium user" \ MW_SERVER=https://en.m.wikipedia.beta.wmflabs.org \ DISPLAY=true \ npm run-script selenium-test
To run an individual test:
$> MEDIAWIKI_PASSWORD={from officewiki} \ MEDIAWIKI_USER="Selenium user" \ MW_SERVER=https://en.m.wikipedia.beta.wmflabs.org \ DISPLAY=true \ npm run-script selenium-test \ -- --spec tests/selenium/specs/category.js
The beta-cluster selenium user password is this officewiki page https://office.wikimedia.org/wiki/Selenium_passwords
Developer notes
Manual testing of the wikitext editor on the beta cluster does not produce any errors, so it's likely the test itself is faulty.It seems the`invalid csrf token` errors has also impacted Watchlist tests[1] as well.
It's possible that the combination of API login + UI login are causing the CSRF tokens to be invalidated. The beforeEach step calls iAmLoggedIntoTheMobileWebsite() which logs in via the login page, but then the "Redirects" step calls pageExists( title );which logs in the same user via the API. When that second API login occurs, the previous login token might be invalidated, causing the "browser" user to be logged out.
It might be worth either:
1.) Avoiding using the API when it's not necessary, and create pages via the UI instead.
2.) Using separate users for API actions and browser actions, so that they don't share the same CSRF token.
A previous attempt at a fixing this was tried in T223676.
[1] https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-daily-beta-Minerva/78/console
Acceptance Criteria
- The daily test job starts passing