Page MenuHomePhabricator

Find out why 'old revisions do not have edit links' selenium test is flaky on CI
Open, MediumPublicPRODUCTION ERROR

Description

As advised in T227121:
Some CI builds have been failing due to this Wikibase selenium test (currently skipped on master)
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/refs/heads/master/repo/tests/selenium/specs/item.js#102

Examples for failed builds (seeme to fail on every second run or so):
https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/29044/console
https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-hhvm-docker/18210/console

The video seems to indicate not being able to click on the REVISION_DATE_LINK defined in repo/tests/selenium/wdio-wikibase/pageobjects/item.page.js

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 520750 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[mediawiki/extensions/Wikibase@master] Fix flaky it block to assert no edit link on old revision

https://gerrit.wikimedia.org/r/520750

Change 520750 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix flaky it block to assert no edit link on old revision

https://gerrit.wikimedia.org/r/520750

Giving it some time for some wikibase builds to pass to ensure this isn't that flaky anymore

@noarave if you haven't started looking into this yet (and not planning soon) I suggest the following:

  • skip this test for now
  • look into the original issue and see if we can write a different (non-browser) test

what do you think?

(I'm thinking to try with this and see how we would approach those cases with tests that are not the usual browser tests we write usually, connected in my head to T226975)

@alaa_wmde i'm fine with this, as a long standing solution seems to be less than obvious by now.

Change 525234 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[mediawiki/extensions/Wikibase@master] Skip flaky wikibase item browser test for now

https://gerrit.wikimedia.org/r/525234

Change 525234 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Skip flaky wikibase item browser test for now

https://gerrit.wikimedia.org/r/525234

I'm really wondering what to do about this, since this test seemed to be failing relatively often on CI but for different reasons than it might be failing locally every once in a while (which it didn't always do but currently does). I think this needs some decision regarding whether it makes sense to test this in any other way (phpunit + Hamcrest) and is this scenario worth this sort of investment. would like opinions @alaa_wmde @WMDE-leszek @Ladsgroup @Rosalie_WMDE

Chiming in here: At Wikidata Bridge we started using the "new" wdio5, instead of the current wdio4. Is it possible that upgrading might improve the situation?

There is a patch in the making to make wdio-mediawiki compatible with wdio 5: https://gerrit.wikimedia.org/r/528882

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:06 PM

The patch should have fixed the issue hopefully. Feel free to bring it back if flaky tests keep happening

Change 534440 had a related patch set uploaded (by Noa wmde; owner: Noa wmde):
[mediawiki/extensions/Wikibase@master] Unskipping test with hopes of flakiness averted

https://gerrit.wikimedia.org/r/534440

Change 534440 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Unskipping test with hopes of flakiness averted

https://gerrit.wikimedia.org/r/534440

Change 535879 had a related patch set uploaded (by Jakob; owner: Jakob):
[mediawiki/extensions/Wikibase@master] Skip browser test checking edit links on old revisions

https://gerrit.wikimedia.org/r/535879

Jakob_WMDE added a subscriber: Jakob_WMDE.

This is still flaky, it seems. I uploaded a patch to skip it again.

Probably worth looking into:

I assume this is caused by the ItemPage.goToPreviousRevision() function, which as far as I can tell is only used by this one test, and looks extremely fragile. Here’s what it does:

  • click on the recent changes link in the sidebar
  • click on the “history” link of the last change in the wiki
  • click on the date link of the change marked as “after” (this class is assigned based on the two diff selection radio buttons – initially, “before” is the current revision and “after” its parent)

I’m not sure what part of this breaks, but since we already know the item ID, at least the first two steps seem unnecessary to me, and for the last step it might also be better to make an API call to get the second-latest revision.

Change 535879 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Skip browser test checking edit links on old revisions

https://gerrit.wikimedia.org/r/535879

Addshore added a subscriber: Addshore.

So, this is still skipped!

Addshore lowered the priority of this task from High to Medium.May 5 2020, 11:55 AM

T232629: Support query parameters in Special:EntityPage was just resolved after being inactive for a few months; I no longer remember the context for that task, but I think it might have been related to this:

This is still flaky, it seems. I uploaded a patch to skip it again.

Probably worth looking into:

I assume this is caused by the ItemPage.goToPreviousRevision() function, which as far as I can tell is only used by this one test, and looks extremely fragile. Here’s what it does:

  • click on the recent changes link in the sidebar
  • click on the “history” link of the last change in the wiki
  • click on the date link of the change marked as “after” (this class is assigned based on the two diff selection radio buttons – initially, “before” is the current revision and “after” its parent)

I’m not sure what part of this breaks, but since we already know the item ID, at least the first two steps seem unnecessary to me, and for the last step it might also be better to make an API call to get the second-latest revision.

It may be possible to replace ItemPage.goToPreviousRevision() with a more robust implementation now that Special:EntityPage/Q123?revision=456 Special:EntityPage/Q123?oldid=456 is supported.

tstarling raised the priority of this task from Medium to High.Fri, May 14, 1:03 AM

Looks like this test was reenabled a couple of days ago, by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/683914 . Since then it's been a constant nuisance, failing on maybe 20% of core patches. I don't understand why the test was re-enabled prior to proposed fix https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/690044/ being merged.

Change 690806 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/Wikibase@master] Revert "Reenable test to check no edit links on old revision"

https://gerrit.wikimedia.org/r/690806

Change 690806 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Revert "Reenable test to check no edit links on old revision"

https://gerrit.wikimedia.org/r/690806

tstarling lowered the priority of this task from High to Medium.Fri, May 14, 3:26 AM

Change 690044 had a related patch set uploaded (by Hoo man; author: Hoo man):

[mediawiki/extensions/Wikibase@master] Re-Add "item old revisions do not have an edit link"

https://gerrit.wikimedia.org/r/690044

Maintenance_bot moved this task from Research to Active on the wdwb-tech board.
Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.