Page MenuHomePhabricator

Daily Minerva Selenium test job failing
Closed, ResolvedPublic

Description

The selenium-daily-beta-Minerva job has been failing since the beginning of May. This job runs the full Minerva selenium test suite targeting the beta cluster on a daily basis. The test logs as well as build artifacts are available on Jenkins .

The first tests failure occurs on the "Wikitext Editor (Makes actual saves)" test suite. There are 3 tests in this file:

tests/selenium/specs/editor_wikitext_saving.js
// @test2.m.wikipedia.org @login
describe( 'Wikitext Editor (Makes actual saves)', () => {

	beforeEach( () => {
		iAmLoggedIntoTheMobileWebsite();
	} );

	// @editing
	it( 'It is possible to edit', () => {
		iGoToAPageThatHasLanguages();
		iClickTheEditButton();
		iSeeTheWikitextEditorOverlay();
		iTypeIntoTheEditor( 'ABC GHI' );
		iClickContinue();
		iClickSubmit();
		iDoNotSeeTheWikitextEditorOverlay();
		iShouldSeeAToastNotification();
	} );

	// @editing @en.m.wikipedia.beta.wmflabs.org
	it( 'Redirects', () => {
		const title = 'Selenium wikitext editor test ' + Math.random();
		pageExists( title );
		iAmOnAPageThatDoesNotExist();
		iClickTheEditButton();
		iSeeTheWikitextEditorOverlay();
		iClearTheEditor();
		iTypeIntoTheEditor( `#REDIRECT [[${title}]]` );
		iClickContinue();
		iClickSubmit();
		iSayOkayInTheConfirmDialog();
		iDoNotSeeTheWikitextEditorOverlay();
		theTextOfTheFirstHeadingShouldBe( title );
	} );

	// @editing @en.m.wikipedia.beta.wmflabs.org
	it( 'Broken redirects', () => {
		iAmOnAPageThatDoesNotExist();
		iClickTheEditButton();
		iSeeTheWikitextEditorOverlay();
		iClearTheEditor();
		iTypeIntoTheEditor( '#REDIRECT [[AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA]]' );
		iClickContinue();
		iClickSubmit();
		iSayOkayInTheConfirmDialog();
		iDoNotSeeTheWikitextEditorOverlay();
		thereShouldBeARedLinkWithText( 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' );
	} );
} );

See https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-daily-beta-Minerva/80/ as an example build that failed. The tests complain of:

  • badtoken: Invalid CSRF token.
  • Could not login: WrongToken
  • element (".overlay .wikitext-editor") still not existing after 10000ms

Reproduction steps

To run the Selenium tests against the beta cluster on your local machine, ensure you have chromedriver installed, then npm install.
Start chromedriver in one tab:

$> chromedriver --url-base=/wd/hub --port=4444

and start the tests in another:

$> MEDIAWIKI_PASSWORD={from officewiki} \
MEDIAWIKI_USER="Selenium user" \
MW_SERVER=https://en.m.wikipedia.beta.wmflabs.org \
DISPLAY=true \
npm run-script selenium-test

To run an individual test:

$> MEDIAWIKI_PASSWORD={from officewiki} \
MEDIAWIKI_USER="Selenium user" \
MW_SERVER=https://en.m.wikipedia.beta.wmflabs.org \
DISPLAY=true \
npm run-script selenium-test \
-- --spec tests/selenium/specs/category.js

The beta-cluster selenium user password is this officewiki page https://office.wikimedia.org/wiki/Selenium_passwords

Developer notes

Manual testing of the wikitext editor on the beta cluster does not produce any errors, so it's likely the test itself is faulty.It seems the`invalid csrf token` errors has also impacted Watchlist tests[1] as well.

It's possible that the combination of API login + UI login are causing the CSRF tokens to be invalidated. The beforeEach step calls iAmLoggedIntoTheMobileWebsite() which logs in via the login page, but then the "Redirects" step calls pageExists( title );which logs in the same user via the API. When that second API login occurs, the previous login token might be invalidated, causing the "browser" user to be logged out.

It might be worth either:
1.) Avoiding using the API when it's not necessary, and create pages via the UI instead.
2.) Using separate users for API actions and browser actions, so that they don't share the same CSRF token.

A previous attempt at a fixing this was tried in T223676.

[1] https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-daily-beta-Minerva/78/console

Acceptance Criteria

  • The daily test job starts passing

Event Timeline

nray created this task.
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptJun 3 2019, 11:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
nray renamed this task from editor_wikitext_saving.js selenium tests are failing. to editor_wikitext_saving.js selenium tests are failing.Jun 3 2019, 11:18 PM
Jdrewniak renamed this task from editor_wikitext_saving.js selenium tests are failing to Daily Minerva Selenium test job failing.Jun 5 2019, 9:51 PM
Jdrewniak updated the task description. (Show Details)
Jdrewniak added subscribers: Jdlrobson, zeljkofilipin.

Wrong token error

I've been able to consistently reproduce a CSFR "wrong token" error with the following code in the Minerva test folder:

wrong-csrf-token.js
const 
	{ iAmLoggedIntoTheMobileWebsite } = require( '../features/step_definitions/common_steps' ),
	{ pageExists } = require( './../features/step_definitions/common_steps' ),
	title = 'Selenium wikitext editor test';

describe( 'csrf-token-invalidation-test', () => {
	beforeEach( () => {
		pageExists( title );
	} );

	it( 'will produce a CSRF error', () => {
		iAmLoggedIntoTheMobileWebsite();
		pageExists( title );
	} );
} );

pageExists() is an API function that logs into the site and attempts to create a page.
iAmLoggedIntoTheMobileWebsite() is a function that logs in through the browser via the login page.

A preliminary Phabricator search reveals that the "wrong token" error has been encountered numerous times for different reasons T160519 T216641 T222517 but I suspect that this flow of logging in via the API, then through the UI, then through the API again causes the token to be invalidated.

Error setting beta mode

For a while, I've been experiencing an error opting into beta mode while running the tests against the beta cluster. Enabling beta-mode is just a matter of setting a cookie, so I suspected that maybe we're setting the beta cookie incorrectly. One thing that might be a problem is that we're not setting an expiry date on the optin cookie. However, even when I added the expiry value, beta mode was still not being activated.

Then I realized that when I ran the tests against the beta cluster, I was using the URL https://en.wikipedia.beta.wmflabs.org. The cookie was being set, but for the wrong domain. The cookie should be set on the mobile domain instead https://en.m.wikipedia.beta.wmflabs.org. Then I remembered this issue being addressed in T220658, which set the daily job to start running from the mobile domain.

This solution however, is inconsistencies with local development environments. On most dev environments, we assume the test starts out in desktop mode, and switch to mobile mode. This switch works locally because developers typically don't use an m-dot domain, and so the mobile cookie is set correctly. On the daily beta job however, this switch doesn't work but the tests still run because they start from a different assumption.

I think it might be easier to write these tests if they started from the same set of assumptions. Also, using the UI to set the mobile mode might be less error prone than setting the cookie.

Change 514694 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/skins/MinervaNeue@master] Ensure each api call creates a new instance of MWBot

https://gerrit.wikimedia.org/r/514694

Wrong token error
I've tracked down our special Minerva token error (I hope). It comes from the World.js file exporting an instance of MWBot, which holds on to its edit token. The wdio-mediawiki library requests new edit tokens for each api action, and so when we try to use that first instance, it's edit token has already been invalidated.

Change 514694 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Ensure each api call creates a new instance of MWBot

https://gerrit.wikimedia.org/r/514694

After looking over the test results over the past couple of days, it seems that the test is now failing intermittently :(

Screen Shot 2019-06-17 at 22.08.32.png (772×668 px, 127 KB)

Build 97

/src/tests/selenium/specs/editor_wikitext_saving.js
21:45:47 [chrome #0-3] Session ID: 5ef9312723bf12083d7986bf1bbba3dc
21:45:47 [chrome #0-3] Spec: /src/tests/selenium/specs/editor_wikitext_saving.js
21:45:47 [chrome #0-3] Running: chrome
21:45:47 [chrome #0-3]
21:45:47 [chrome #0-3] Wikitext Editor (Makes actual saves)
21:45:47 [chrome #0-3]   ✓ It is possible to edit
21:45:47 [chrome #0-3]   ✓ Redirects
21:45:47 [chrome #0-3]   1) "before each" hook for "Broken redirects"
21:45:47 [chrome #0-3]
21:45:47 [chrome #0-3]
21:45:47 [chrome #0-3] 2 passing (24s)
21:45:47 [chrome #0-3] 1 failing
21:45:47 [chrome #0-3]
21:45:47 [chrome #0-3] 1) Wikitext Editor (Makes actual saves) "before each" hook for "Broken redirects":
21:45:47 [chrome #0-3] Input A expected to strictly equal input B:
21:45:47 + expected - actual
21:45:47 
21:45:47 - false
21:45:47 + true
21:45:47 
21:45:47 [chrome #0-3] AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:
21:45:47 [chrome #0-3] + expected - actual
21:45:47 [chrome #0-3] 
21:45:47 [chrome #0-3] - false
21:45:47 [chrome #0-3] + true
21:45:47 [chrome #0-3]     at iAmLoggedIn (/src/tests/selenium/features/step_definitions/common_steps.js:57:9)
21:45:47 [chrome #0-3]     at iAmLoggedIntoTheMobileWebsite (/src/tests/selenium/features/step_definitions/common_steps.js:62:2)
21:45:47 [chrome #0-3]     at Context.beforeEach (/src/tests/selenium/specs/editor_wikitext_saving.js:17:3)
21:45:47 [chrome #0-3]     at new Promise (<anonymous>)
21:45:47 [chrome #0-3]     at new F (/src/node_modules/core-js/library/modules/_export.js:36:28)
21:45:47 [chrome #0-3]
21:45:47

Broken-redirects.png (888×1 px, 33 KB)

Build 96

/src/tests/selenium/specs/editor_wikitext_saving.js
21:43:12 [chrome #0-3] Session ID: db27aefcbd6ddae754022447fabd860b
21:43:12 [chrome #0-3] Spec: /src/tests/selenium/specs/editor_wikitext_saving.js
21:43:12 [chrome #0-3] Running: chrome
21:43:12 [chrome #0-3]
21:43:12 [chrome #0-3] Wikitext Editor (Makes actual saves)
21:43:12 [chrome #0-3]   1) It is possible to edit
21:43:12 [chrome #0-3]   2) "before each" hook for "Redirects"
21:43:12 [chrome #0-3]
21:43:12 [chrome #0-3]
21:43:12 [chrome #0-3] 2 failing (20s)
21:43:12 [chrome #0-3]
21:43:12 [chrome #0-3] 1) Wikitext Editor (Makes actual saves) It is possible to edit:
21:43:12 [chrome #0-3] Promise was rejected with the following reason: timeout
21:43:12 [chrome #0-3] Error: Promise was rejected with the following reason: timeout
21:43:12 [chrome #0-3]     at elementIdDisplayed("0.9716392571999009-3") - isVisible.js:71:55
21:43:12 [chrome #0-3]
21:43:12 [chrome #0-3] 2) Wikitext Editor (Makes actual saves) "before each" hook for "Redirects":
21:43:12 [chrome #0-3] unexpected alert open: {Alert text : }
21:43:12 [chrome #0-3] Error: A modal dialog was open, blocking this operation
21:43:12 [chrome #0-3]     at url("https://en.m.wikipedia.beta.wmflabs.org/w/index.php?title=Special%3AUserLogin") - index.js:312:3
21:43:12 [chrome #0-3]

It-is-possible-to-edit.png (888×1 px, 31 KB)

Build 95

/src/tests/selenium/specs/watchstar.js
21:36:37 ------------------------------------------------------------------
21:36:37 [chrome #0-9] Session ID: 63d4a455f84727d67564d3acf589a128
21:36:37 [chrome #0-9] Spec: /src/tests/selenium/specs/watchstar.js
21:36:37 [chrome #0-9] Running: chrome
21:36:37 [chrome #0-9]
21:36:37 [chrome #0-9] Manage Watchlist
21:36:37 [chrome #0-9]   1) Remove an article from the watchlist
21:36:37 [chrome #0-9]   ✓ Add an article to the watchlist
21:36:37 [chrome #0-9]
21:36:37 [chrome #0-9]
21:36:37 [chrome #0-9] 1 passing (20s)
21:36:37 [chrome #0-9] 1 failing
21:36:37 [chrome #0-9]
21:36:37 [chrome #0-9] 1) Manage Watchlist Remove an article from the watchlist:
21:36:37 [chrome #0-9] Input A expected to strictly equal input B:
21:36:37 + expected - actual
21:36:37 
21:36:37 - false
21:36:37 + true
21:36:37 [chrome #0-9] AssertionError [ERR_ASSERTION]: Input A expected to strictly equal input B:
21:36:37 [chrome #0-9] + expected - actual
21:36:37 [chrome #0-9] 
21:36:37 [chrome #0-9] - false
21:36:37 [chrome #0-9] + true
21:36:37 [chrome #0-9]     at iShouldSeeAToastNotificationWithMessage (/src/tests/selenium/features/step_definitions/common_steps.js:84:9)
21:36:37 [chrome #0-9]     at Context.it (/src/tests/selenium/specs/watchstar.js:21:3)
21:36:37 [chrome #0-9]     at new Promise (<anonymous>)
21:36:37 [chrome #0-9]     at new F (/src/node_modules/core-js/library/modules/_export.js:36:28)
21:36:37 [chrome #0-9]

Remove-an-article-from-the-watchlist.png (888×1 px, 60 KB)

Build 94

/src/tests/selenium/specs/editor_wikitext_saving.js
21:45:48 [chrome #0-3] Session ID: 489a097bdc220ad17c1c0f95c0ff25cf
21:45:48 [chrome #0-3] Spec: /src/tests/selenium/specs/editor_wikitext_saving.js
21:45:48 [chrome #0-3] Running: chrome
21:45:48 [chrome #0-3]
21:45:48 [chrome #0-3] Wikitext Editor (Makes actual saves)
21:45:48 [chrome #0-3]   1) It is possible to edit
21:45:48 [chrome #0-3]   2) "before each" hook for "Redirects"
21:45:48 [chrome #0-3]
21:45:48 [chrome #0-3]
21:45:48 [chrome #0-3] 2 failing (20s)
21:45:48 [chrome #0-3]
21:45:48 [chrome #0-3] 1) Wikitext Editor (Makes actual saves) It is possible to edit:
21:45:48 [chrome #0-3] Promise was rejected with the following reason: timeout
21:45:48 [chrome #0-3] Error: Promise was rejected with the following reason: timeout
21:45:48 [chrome #0-3]     at elementIdDisplayed("0.78719815492501-4") - isVisible.js:71:55
21:45:48 [chrome #0-3]
21:45:48 [chrome #0-3] 2) Wikitext Editor (Makes actual saves) "before each" hook for "Redirects":
21:45:48 [chrome #0-3] unexpected alert open: {Alert text : }
21:45:48 [chrome #0-3] Error: A modal dialog was open, blocking this operation
21:45:48 [chrome #0-3]     at url("https://en.m.wikipedia.beta.wmflabs.org/w/index.php?title=Special%3AUserLogin") - index.js:312:3
21:45:48 [chrome #0-3]
21:45:48

It-is-possible-to-edit-1.png (888×1 px, 31 KB)

Build 91

title=" /src/tests/selenium/specs/editor_wikitext_saving.js", lines=12
21:45:50 [chrome #0-3] Session ID: ea0103de801d46debfef9033eebb255b
21:45:50 [chrome #0-3] Spec: /src/tests/selenium/specs/editor_wikitext_saving.js
21:45:50 [chrome #0-3] Running: chrome
21:45:50 [chrome #0-3]
21:45:50 [chrome #0-3] Wikitext Editor (Makes actual saves)
21:45:50 [chrome #0-3]   1) It is possible to edit
21:45:50 [chrome #0-3]   2) "before each" hook for "Redirects"
21:45:50 [chrome #0-3]
21:45:50 [chrome #0-3]
21:45:50 [chrome #0-3] 2 failing (19s)
21:45:50 [chrome #0-3]
21:45:50 [chrome #0-3] 1) Wikitext Editor (Makes actual saves) It is possible to edit:
21:45:50 [chrome #0-3] Promise was rejected with the following reason: timeout
21:45:50 [chrome #0-3] Error: Promise was rejected with the following reason: timeout
21:45:50 [chrome #0-3]     at elementIdDisplayed("0.2699168560876135-4") - isVisible.js:71:55
21:45:50 [chrome #0-3]
21:45:50 [chrome #0-3] 2) Wikitext Editor (Makes actual saves) "before each" hook for "Redirects":
21:45:50 [chrome #0-3] unexpected alert open: {Alert text : }
21:45:50 [chrome #0-3] Error: A modal dialog was open, blocking this operation
21:45:50 [chrome #0-3]     at url("https://en.m.wikipedia.beta.wmflabs.org/w/index.php?title=Special%3AUserLogin") - index.js:312:3
21:45:50 [chrome #0-3]
21:45:50

It-is-possible-to-edit-2.png (888×1 px, 31 KB)

It looks like the flakiest test is editor_wikitext_saving.js. The error message in the screenshot "Error: Another user had edited this page" suggests some sort of race condition or async error.

Change 520133 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Browser: Drop failing tests

https://gerrit.wikimedia.org/r/520133

Change 520133 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Browser: Drop failing tests

https://gerrit.wikimedia.org/r/520133

@Jdlrobson I have the tests working on my local machine.

The daily builds are green for now. Next up T227073

Change 542619 had a related patch set uploaded (by Edward Tadros; owner: Edward Tadros):
[mediawiki/skins/MinervaNeue@master] WIP: Temporarily fixed waitForPropogation.

https://gerrit.wikimedia.org/r/542619

Change 542619 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Temporarily fixed waitForPropogation.

https://gerrit.wikimedia.org/r/542619