Page MenuHomePhabricator

Cite's cypress browser test "Cite backlinks test" is flaky
Closed, ResolvedPublic

Description

On gate-and-submit for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/992230 a cypress failure was seen

New (or ported) tests from 0f801ea5

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/78610/console
video for 7 days at https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php74-docker/78610/artifact/log/Cite/backlinks.cy.js.mp4

5 passing (31s)
1 failing

1) Cite backlinks test
     highlights backlink in the reference list for the clicked reference when there are multiple used references:
   AssertionError: Timed out retrying after 4000ms: expected '<a>' to have class 'mw-cite-targeted-backlink'
    at Context.eval (webpack://Cite/./cypress/e2e/tests/backlinks.cy.js:59:34)

Event Timeline

awight triaged this task as High priority.Jan 24 2024, 2:09 PM

Increasing priority since this flakiness is blocking patches on other repositories.

Jdlrobson raised the priority of this task from High to Unbreak Now!.Jan 30 2024, 12:19 AM
Jdlrobson subscribed.

I have a train blocker that I can't currently merge due to this issue.

This was supposed to have un-flaked the tests: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Cite/+/991532

It's fine to skip them, if they're still broken of course.

I've confirmed that the tests are still flaky, affecting patch https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/993808 . CI had checked out the master branch of Cite which included the improvements to avoid race conditions.

It seems there is another race condition to guard against, but I don't fully understand yet. This time the failure was while loading the newly-created test page:

00:06:37.108   1) Cite backlinks test
00:06:37.108        "before each" hook for "hides clickable up arrow by default when there are multiple backlinks":
00:06:37.108      CypressError: `cy.visit()` failed trying to load:
00:06:37.108 
00:06:37.108 http://127.0.0.1:9413/index.php?title=CiteTest-title0.27191064498966355
00:06:37.108 
00:06:37.109 The response we received from your web server was:
00:06:37.126 
00:06:37.126   > 404: Not Found
00:06:37.126 
00:06:37.126 This was considered a failure because the status code was not `2xx`.

...

00:06:37.137   From Your Spec Code:
00:06:37.137       at Context.eval (webpack://Cite/./cypress/e2e/tests/backlinks.cy.js:29:5)

The video shows that the page doesn't load at all--it's not the MediaWiki 404 page.
Video: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php81-docker/1766/artifact/log/Cite/backlinks.cy.js.mp4

Update: the 404 page is being intercepted at some level so getting the default browser 404 is unsurprising. My current theory is that the page is not fully created so I've added an await and an assertion on the API response.

Change 994010 had a related patch set uploaded (by Awight; author: Awight):

[mediawiki/extensions/Cite@master] Verify that the test page was created

https://gerrit.wikimedia.org/r/994010

Change 994010 merged by jenkins-bot:

[mediawiki/extensions/Cite@master] Verify that the test page was created

https://gerrit.wikimedia.org/r/994010

Tobi_WMDE_SW subscribed.

should be fixed now. the team will keep an eye on the tests for some more days though.

thiemowmde renamed this task from Cite's cypress browser test "Cite backlinks test"is flaky to Cite's cypress browser test "Cite backlinks test" is flaky.Feb 15 2024, 11:02 AM
thiemowmde lowered the priority of this task from Unbreak Now! to Needs Triage.