Wikibase browser tests are flaky
Closed, InvalidPublic
Actions

Assigned To

Authored By

	Tarrow
	Oct 30 2020, 11:02 AM

Project Tags

Referenced Files

None

Subscribers

Description

We've recently seen substantially flaky browsers tests:
c.f.
https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-selenium-docker/11727/console

Mostly seems to be due to slow saving.

00:10:35.328 [Chrome 73.0.3683.75 linux #0-1] 1) item can add a statement using the keyboard
00:10:35.328 [Chrome 73.0.3683.75 linux #0-1] element (".valueview-input") still existing after 10000ms
00:10:35.328 [Chrome 73.0.3683.75 linux #0-1] Error: element (".valueview-input") still existing after 10000ms
00:10:35.328 [Chrome 73.0.3683.75 linux #0-1]     at Context.<anonymous> (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/item.js:65:28)

We should try to make these less fragile (maybe increase possible timeouts)

Related Objects

Mentioned In: T265976: Lexeme daily browser tests (against beta) flaky [timebox 8h]
Mentioned Here: T265596: Investigate and possibly improve stability of browser tests

Event Timeline

Tarrow created this task.Oct 30 2020, 11:02 AM

Restricted Application added a project: Wikidata. · View Herald TranscriptOct 30 2020, 11:02 AM

Restricted Application added a subscriber: • Aklapper. · View Herald Transcript

might be worth mentioning this here as well -> https://phabricator.wikimedia.org/T265596

Michael subscribed.Oct 30 2020, 1:54 PM

Addshore mentioned this in T265976: Lexeme daily browser tests (against beta) flaky [timebox 8h].Nov 6 2020, 1:02 PM

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Nov 6 2020, 1:15 PM

We struggled to determine the cause of this. As speculated it could be due to some network or serverside failure. Digging through the server side debug logs didn't show any very obvious things though.

We considered that it is unfortunate that we can't see the normal brower console logs since this might show us failures such as load.php requests failing succeed.

Sadly it looks like the ability to do this isn't there in the newer webdriver spec. See: https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidlog (the old version) VS https://w3c.github.io/webdriver/

Since it seems to have magically fixed itself; it couldn't be reproduced locally; and we don't have a great strategy for making ourselves better able to cope with the flaky tests in the future I'll close as invalid.

Wikibase browser tests are flakyClosed, InvalidPublicActions

Description

Related Objects

Event Timeline

Wikibase browser tests are flaky
Closed, InvalidPublic
Actions