Many MinervaNeue browser tests are failing intermittently but often on Chrome and Firefox
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• Niedzielski
	Nov 17 2017, 6:25 PM

Description

https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/202/
https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/203/

Some of the tests that are failing are listed below (the actual tests failing are changing across runs).

Chrome:

Talk.Talk on a page that does exist
Menus open correct page for anonymous users.Check links in menu
Menus open correct page for anonymous users.Check links in menu
Search.Going back to the previous page
Language selection.Tapping icon does not open language overlay (on a page that doesn't have languages)

Firefox:

Table of contents.Show table of contents on tablet
Wikidata descriptions.Description appears on main namespace
Wikidata descriptions.Description does not appear on non-main namespaces
Search.Clicking on a watchstar toggles the watchstar
Search.Going back to the previous page
Search.Search doesn't break after one search result
Page actions menu when anonymous.I cannot edit a protected page when anonymous
Page diff.Added and removed content
Wikitext Editor.Closing editor (overlay button)
Notification.Opening notifications
Basic site for legacy devices.Nearby link not present in main navigation menu
Talk.Talk on a page that doesn't exist (bug 64268)
Language selection.Language button (on a page that doesn't have languages)
Language selection.Checking that there are no suggested language links
Signup edit tutorial.Signup edit tutorial shows up correctly and hides when main menu is opened
Toggling sections.Opening a section on mobile
Toggling sections.Closing a section on mobile
Search.Closing search (browser button)
Talk.Add discussion for talk page possible as logged in user
Language selection.Tapping icon opens language overlay
Language selection.Tapping the disabled icon shows a toast
Wikitext Editor (Makes actual saves).Redirects

Possibly related to: T176316

Related Objects

Mentioned Here: T174018: Port Minerva's 7 Ruby-based browser tests to Selenium Node.js
T176316: Automated browser tests cannot create pages on the Beta Cluster

Event Timeline

• Niedzielski created this task.Nov 17 2017, 6:25 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 17 2017, 6:25 PM

This has always been the case and I suspect it is because of the sheer amount of tests that are run. The probability of one failing is thus higher. In addition to this, the success of the tests relies on the stability of the beta cluster and Saucelabs.

In the builds you mention, if you drill down into the individual scenarios you'll see " Uh-oh. Could not find link to Sauce Labs job URL. Net::ReadTimeout (Net::ReadTimeout)", "Uh-oh. Could not find link to Sauce Labs job URL. unexpected response, code=404, content-type="text/plain""

there's been a longstanding bugs about this (https://phabricator.wikimedia.org/search/query/evYIiCkqo4QO/#R)

It looks like this time there was an issue with SauceLabs.

I'm not sure if when T174018 is complete they will be more stable, but I've re-run the build now and suspect it will pass.
I'm not sure if there are any other actions we can take here.

@zeljkofilipin @hashar the failures are a little unusual. Did anything change with our configuration recently?

Sure enough, the tests have recovered over the weekend. I don't suggest investigating this more. Let's focus on getting them ported to Node.js

Many MinervaNeue browser tests are failing intermittently but often on Chrome and FirefoxClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

Many MinervaNeue browser tests are failing intermittently but often on Chrome and Firefox
Closed, DeclinedPublic
Actions