Page MenuHomePhabricator

Find out why 'old revisions do not have edit links' selenium test is flaky on CI
Open, HighPublic

Description

As advised in T227121:
Some CI builds have been failing due to this Wikibase selenium test (currently skipped on master)
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/refs/heads/master/repo/tests/selenium/specs/item.js#102

Examples for failed builds (seeme to fail on every second run or so):
https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/29044/console
https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-hhvm-docker/18210/console

The video seems to indicate not being able to click on the REVISION_DATE_LINK defined in repo/tests/selenium/wdio-wikibase/pageobjects/item.page.js

Details

Related Gerrit Patches:
mediawiki/extensions/Wikibase : masterSkip browser test checking edit links on old revisions
mediawiki/extensions/Wikibase : masterUnskipping test with hopes of flakiness averted
mediawiki/extensions/Wikibase : masterSkip flaky wikibase item browser test for now
mediawiki/extensions/Wikibase : masterFix flaky it block to assert no edit link on old revision

Event Timeline

noarave created this task.Jul 4 2019, 10:39 AM
Restricted Application added a project: User-Noarave. · View Herald TranscriptJul 4 2019, 10:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
noarave updated the task description. (Show Details)Jul 4 2019, 12:25 PM

Change 520750 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[mediawiki/extensions/Wikibase@master] Fix flaky it block to assert no edit link on old revision

https://gerrit.wikimedia.org/r/520750

Change 520750 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix flaky it block to assert no edit link on old revision

https://gerrit.wikimedia.org/r/520750

Giving it some time for some wikibase builds to pass to ensure this isn't that flaky anymore

alaa_wmde added a comment.EditedJul 23 2019, 1:19 PM

@noarave if you haven't started looking into this yet (and not planning soon) I suggest the following:

  • skip this test for now
  • look into the original issue and see if we can write a different (non-browser) test

what do you think?

(I'm thinking to try with this and see how we would approach those cases with tests that are not the usual browser tests we write usually, connected in my head to T226975)

@alaa_wmde i'm fine with this, as a long standing solution seems to be less than obvious by now.

Change 525234 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[mediawiki/extensions/Wikibase@master] Skip flaky wikibase item browser test for now

https://gerrit.wikimedia.org/r/525234

Change 525234 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Skip flaky wikibase item browser test for now

https://gerrit.wikimedia.org/r/525234

I'm really wondering what to do about this, since this test seemed to be failing relatively often on CI but for different reasons than it might be failing locally every once in a while (which it didn't always do but currently does). I think this needs some decision regarding whether it makes sense to test this in any other way (phpunit + Hamcrest) and is this scenario worth this sort of investment. would like opinions @alaa_wmde @WMDE-leszek @Ladsgroup @Rosalie_WMDE

Chiming in here: At Wikidata Bridge we started using the "new" wdio5, instead of the current wdio4. Is it possible that upgrading might improve the situation?

There is a patch in the making to make wdio-mediawiki compatible with wdio 5: https://gerrit.wikimedia.org/r/528882

noarave removed noarave as the assignee of this task.Aug 27 2019, 8:34 AM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM
Ladsgroup claimed this task.Sep 3 2019, 4:25 PM

The patch should have fixed the issue hopefully. Feel free to bring it back if flaky tests keep happening

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptSep 3 2019, 4:25 PM

Change 534440 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[mediawiki/extensions/Wikibase@master] Unskipping test with hopes of flakiness averted

https://gerrit.wikimedia.org/r/534440

Change 534440 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Unskipping test with hopes of flakiness averted

https://gerrit.wikimedia.org/r/534440

Change 535879 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Skip browser test checking edit links on old revisions

https://gerrit.wikimedia.org/r/535879

Jakob_WMDE added a subscriber: Jakob_WMDE.

This is still flaky, it seems. I uploaded a patch to skip it again.

Probably worth looking into:

I assume this is caused by the ItemPage.goToPreviousRevision() function, which as far as I can tell is only used by this one test, and looks extremely fragile. Here’s what it does:

  • click on the recent changes link in the sidebar
  • click on the “history” link of the last change in the wiki
  • click on the date link of the change marked as “after” (this class is assigned based on the two diff selection radio buttons – initially, “before” is the current revision and “after” its parent)

I’m not sure what part of this breaks, but since we already know the item ID, at least the first two steps seem unnecessary to me, and for the last step it might also be better to make an API call to get the second-latest revision.

Change 535879 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Skip browser test checking edit links on old revisions

https://gerrit.wikimedia.org/r/535879

Ladsgroup removed Ladsgroup as the assignee of this task.Sep 11 2019, 7:01 PM