Page MenuHomePhabricator

Flaky wdio test "RecentChanges shows lemmas in title links"
Closed, ResolvedPublicPRODUCTION ERROR

Description

1) Special:RecentChanges shows lemmas in title links to lexemes:
false == true
running chrome
AssertionError: false == true
    at Context.it (/workspace/src/extensions/WikibaseLexeme/tests/selenium/specs/special/recentchanges.js:38:3)
    at Promise.F (/workspace/src/node_modules/core-js/library/modules/_export.js:36:28)

Event Timeline

Krinkle created this task.Mar 29 2019, 2:33 AM

I believe this might be a similar issue to what was earlier described in T199446?

Krinkle raised the priority of this task from High to Unbreak Now!.Apr 9 2019, 10:57 PM
Krinkle added a project: ContentTranslation.
Krinkle added subscribers: KartikMistry, greg, Nikerabbit and 3 others.

Is there reason to believe this is a CI-Infra issue?

No, I don't think so :)

Right, this basically brings us back to the discussion that was happening in T199446 and in the related patch experiments done by some WMDE engineers.
To take the recent failure (T220229) as an example, here's what I believe happens:

  1. Edit is made with the API (namely, new lexeme is created)
  2. Once done, Special:RecentChanges is opened
  3. Driver is apparently too fast, as the just made edit has not made it to Recent Changes yet.

I see no real way to get around this, as, for valid reasons, Recent changes are populated asynchronously, hence it always be possible, that the recent changes page will be opened before the job updating recent changes has added data from recent edits.
Rather dumb approach I could think of would be to have test wait some arbitrary amount of time between making an edit, and opening the special page, but this does not seem like a sustainable approach. Things could get too slow and this is difficult to predict. Also, adding a "sleep" to all tests dealing with recent changes etc would likely make the duration of the test suite run even longer, while it does quite a significant amount of time already. I'd see this as a (rather annoying) hack that hopefully works for some time, but not really a solution.

We could also ignore testing integration of our tools with Recent Changes, but as Recent Changes is the critical part of wikis, and integral part of editor work flows, ignoring those parts of the system from automated testing seems wrong, to put mildly.

Any ideas how to deal with this?

Please disable the test and figure it out later :)

Change 502827 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/WikibaseLexeme@master] Temporarily skip a flaky selenium test

https://gerrit.wikimedia.org/r/502827

Change 502827 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Temporarily skip a flaky selenium test

https://gerrit.wikimedia.org/r/502827

Change 502830 had a related patch set uploaded (by Pablo Grass (WMDE); owner: Pablo Grass (WMDE)):
[mediawiki/extensions/WikibaseLexeme@master] RecentChanges: run jobs before trying to assert

https://gerrit.wikimedia.org/r/502830

WMDE-leszek lowered the priority of this task from Unbreak Now! to High.Apr 11 2019, 12:11 PM

Change 502830 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] RecentChanges: run jobs before trying to assert

https://gerrit.wikimedia.org/r/502830

WMDE-leszek closed this task as Resolved.Apr 11 2019, 4:11 PM
WMDE-leszek claimed this task.

With https://gerrit.wikimedia.org/r/502830 merged we expect the test to no longer be flaky.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM