If I understand the video correctly, MediaWiki is showing a warning about us saving an empty page, whereas we’re expecting it to save immediately. Is the warning new?

No, wait, we’re supposed to be typing [[${ itemTitle }]] into the text area. Maybe the wikitext editor initializing itself kills our input…

Still seen

• https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1151786
• https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81/18592/console#console-section-9

[0-0] PASSED in chrome - /repo/tests/selenium/specs/blocked.js
[0-1] RUNNING in chrome - /repo/tests/selenium/specs/item.js
[0-1] Error in "item.has its label not rendered when linked on a Wikipage"
Error: Failed to wait for mediawiki.base to be ready after 5000 ms.
    at async Object.waitForModuleState (/workspace/src/extensions/Wikibase/node_modules/wdio-mediawiki/Util.js:50:3)
    at async EntityPage.open (/workspace/src/extensions/Wikibase/node_modules/wdio-wikibase/pageobjects/entity.page.js:9:3)
    at async Context.<anonymous> (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/item.js:93:3)
[0-1] RETRYING in chrome - /repo/tests/selenium/specs/item.js
[0-1] RUNNING in chrome - /repo/tests/selenium/specs/item.js
[0-1] Error in "item.has its label not rendered when linked on a Wikipage"
Error: Timeout of 60000ms exceeded. The execution in the test "item has its label not rendered when linked on a Wikipage" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/item.js)

Related core failure: T380061: Flaky selenium test "Failed to wait for mediawiki.base to be ready" (tests/selenium/specs/page.js)

Change #1190236 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/Wikibase@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/1190236

Seen again here https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/1215/consoleFull

Let's please disable this and then re-enable when we have more confidence that it will not be flaky.

In T388228#10613179, @Lucas_Werkmeister_WMDE wrote:

No, wait, we’re supposed to be typing [[${ itemTitle }]] into the text area. Maybe the wikitext editor initializing itself kills our input…

I tried this out locally now and can’t reproduce it – as far as I can tell, WikiEditor preserves the text area and its value when it initializes itself. Also, neither in that video nor in the video from the build @kostajh just linked do I actually see the expected value being typed, even for a moment.

I wonder if we’re grabbing a reference to #wpTextbox1 just before WikiEditor replaces it, and then type into the old element that’s no longer even part of the document? Is that possible with WDIO?

In T388228#11201497, @Lucas_Werkmeister_WMDE wrote:

I wonder if we’re grabbing a reference to #wpTextbox1 just before WikiEditor replaces it, and then type into the old element that’s no longer even part of the document? Is that possible with WDIO?

I think it is in principle possible (the documentation and linked standard sound like nodes are referenced by identity, not by their selector), but I don’t think it happens here – AFAICT WikiEditor doesn’t replace the original element, it just updates and modifies it.

Change #1190286 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] DNM: Run flaky test a bunch of times to reproduce the failure

https://gerrit.wikimedia.org/r/1190286

Change #1190287 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Try setting the value a second time in browser test

https://gerrit.wikimedia.org/r/1190287

Change #1190286 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] DNM: Run flaky test a bunch of times to reproduce the failure

Reason:

for future reference: according to [this comment](https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1190236/1#message-93f0ed4e3224856ca285c306fff1d55eefaa4fcc), 100 repetitions would’ve been a more reasonable number rather than 1000

https://gerrit.wikimedia.org/r/1190286

Change #1190287 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Try setting the value a second time in browser test

https://gerrit.wikimedia.org/r/1190287

To clarify: the change above tried to fix specifically the following error:

Expect $(#mw-content-text .mw-content-ltr p) to have text

If anyone still sees that error, please comment here.

Independently of that, there also seems to be an unusual amount of “Failed to wait for mediawiki.base to be ready after 5000 ms.” errors happening to this test in particular (“item has its label not rendered when linked on a Wikipage” – e.g. in this build just now). That error sounds like a common error that could in principle affect any browser test; it’s also tracked at T380061: Flaky selenium test "Failed to wait for mediawiki.base to be ready" (tests/selenium/specs/page.js). I don’t understand why it seems to happen so frequently for the “item has its label not rendered” test, and I’m not yet sure if it should be considered part of this task or not.

Seen again today https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/3717/console

Happened again, though didn't fail on a repeat https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/3717/console:

14:39:19 [0-1] Error in "item.has its label not rendered when linked on a Wikipage"
14:39:19 Error: Failed to wait for mediawiki.base to be ready after 5000 ms.
14:39:19     at async Object.waitForModuleState (/workspace/src/extensions/Wikibase/node_modules/wdio-mediawiki/Util.js:50:3)
14:39:19     at async EntityPage.open (/workspace/src/extensions/Wikibase/node_modules/wdio-wikibase/pageobjects/entity.page.js:9:3)
14:39:19     at async Context.<anonymous> (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/item.js:93:3)

In T388228#11229561, @Lucas_Werkmeister_WMDE wrote:

Independently of that, there also seems to be an unusual amount of “Failed to wait for mediawiki.base to be ready after 5000 ms.” errors happening to this test in particular (“item has its label not rendered when linked on a Wikipage” – e.g. in this build just now). That error sounds like a common error that could in principle affect any browser test; it’s also tracked at T380061: Flaky selenium test "Failed to wait for mediawiki.base to be ready" (tests/selenium/specs/page.js). I don’t understand why it seems to happen so frequently for the “item has its label not rendered” test, and I’m not yet sure if it should be considered part of this task or not.

Speculation: the 5000 ms wait is not just for mediawiki.base to be ready, but also for MediaWiki to start sending the page to begin with. If the test just edited the page, then it’s not in the parser cache; I guess it’s possible that the page is expensive enough to parse that this, plus loading JavaScript, doesn’t finish within 5000 ms.

Does anyone know if browser test wikis have a working parser cache? Maybe we could warm up the cache by calling (and awaiting) the action=parse API before issuing the call to open the entity page? Otherwise I guess we can try to await the page load differently, and/or bump the timeout on the waitForModuleState() call.

In T388228#11246126, @Lucas_Werkmeister_WMDE wrote:

Speculation: the 5000 ms wait is not just for mediawiki.base to be ready, but also for MediaWiki to start sending the page to begin with.

No, that doesn’t really make sense, Page.openTitle() already waits for document.readyState === 'complete' (with a 10 s timeout). So “the document and all sub-resources have finished loading” already, yet it still takes more than five seconds to load the ResourceLoader mediawiki.base module?

Okay, a couple of observations:

• There are only three browser tests that use EntityPage.open() (all in item.js). That’s rather fewer than I expected, which makes it somewhat less statistically surprising that we should see the same test name fail so often.
• The other two tests (“can add a statement using the keyboard” and “old revisions do not have an edit link”) both await something else between calling WikibaseApi.createItem() and EntityPage.open() (they get a property and an item, respectively). Conceivably this gives the backend some more time to be ready for the EntityPage.open() call, which is why those tests don’t time out so often?
• There’s a separate method, ItemPage.open(), used by readmode.references.js. This one is just a straight wrapper around Page.openTitle( Special:EntityPage/${entityId} ), without waiting for any ResourceLoader modules.
• The “has its label not rendered when linked on a Wikipage” test doesn’t actually need to interact with the entity page at all – it only loads the item page in order to get the necessary mw.config information to construct the title of the item’s talk page.

So let’s switch that test to ItemPage.open() and see if that improves things.

Change #1193900 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Use ItemPage.open() in flaky Selenium test

https://gerrit.wikimedia.org/r/1193900

In T388228#11246899, @Lucas_Werkmeister_WMDE wrote:

• The “has its label not rendered when linked on a Wikipage” test doesn’t actually need to interact with the entity page at all – it only loads the item page in order to get the necessary mw.config information to construct the title of the item’s talk page.

…but some of the variables we use – specifically wgFormattedNamespaces – are only part of the mediawiki.base ResourceLoader module (via ResourceLoader::getSiteConfigSettings()) and not available as early as other variables. TIL.

Then I guess we should just wait longer for the module to become ready?

Change #1193900 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] Use ItemPage.open() in flaky Selenium test

Reason:

incorrect approach, see T388228#11248950

https://gerrit.wikimedia.org/r/1193900

Attempted pull request here (though it looks like CI needs fixing first): https://github.com/wmde/wdio-wikibase/pull/78

Important note: Wikibase is using an outdated version of wdio-mediawiki (T406844: wdio-wikibase depends on wdio-mediawiki v2); anyone investigating this task should make sure to look at the correct version of the wdio-mediawiki code.

(Though that doesn’t make the task less confusing for me. In fact to me the wdio-mediawiki v5 version of waitForModuleState() seems more broken than the v2 version: see Gerrit comment. I was wrong and wdio-mediawiki v5’s waitForModuleState() should work correctly after all.)

Change #1196491 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Update wdio-mediawiki (5.1) and wdio-wikibase (6.1)

https://gerrit.wikimedia.org/r/1196491

This failed three times out of four for me just now, for example at https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php81/6170/console . And this is despite the automatic retry. The test needs to be skipped.

Change #1196787 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/extensions/Wikibase@master] Skip flaky Selenium label test

https://gerrit.wikimedia.org/r/1196787

Change #1196787 abandoned by Tim Starling:

[mediawiki/extensions/Wikibase@master] Skip flaky Selenium label test

Reason:

duplicate of Ia522582f06464a817b743504368cba908fef6b44

https://gerrit.wikimedia.org/r/1196787

Change #1190236 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/1190236

Bringing the conversation into Phabricator from Gerrit, so we can have a centralized discussion.

As I understand it:

• @Lucas_Werkmeister_WMDE and Wikibase engineers want to have this test enabled, because it is providing value in guarding against regressions
• Non-Wikibase engineers are observing significant disruption to CI builds on projects that are unrelated to Wikibase, for code that doesn't relate to this test
• Some amount of flakiness is inevitable in Selenium tests, but this particular test seems to be above the typical flakiness threshold

Quality-and-Test-Engineering-Team folks, do you have data to show 1) how often Selenium builds fail and 2) which tests are responsible for which percentage of failures?

My proposal would be that the test stays disabled until we are able to demonstrate that it is not going to fail in a flaky way. That would require having some data, though.

And to expand on what I just wrote over on Gerrit (also in the interest of a centralized discussion): I am at this point pretty annoyed that the only suggestion everyone else seems to have for this task is to disable the test. I’ve been trying for weeks to figure out why the test has been flaky (and, unless someone has seen the error from T388228#11229561 come back after all, successfully eliminated one of the sources of flakiness); I’ve tried to think out loud, speculate what might be happening, wonder how WebdriverIO does or doesn’t behave, ask questions, all the while receiving zero responses from all the other people who ought to have a shared interest in making our tests less flaky. It is very frustrating to then be accused of not doing enough about this flaky test. We finally made some more progress again a few days ago when @hoo kindly merged some GitHub pull requests in wdio-wikibase trying to improve the situation; my hope is that upgrading wdio-wikibase (and thus wdio-mediawiki) will help with the flakiness, but T406844: wdio-wikibase depends on wdio-mediawiki v2 has run into some issues of its own where others could potentially be of assistance (especially if anyone’s seen those errors before).

In T388228#11283920, @gerritbot wrote:

Change #1190236 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] selenium: Skip flaky test

https://gerrit.wikimedia.org/r/1190236

(Also, for the record, I submitted and merged a revert of this change, it just wasn’t attached to this task.)

@Lucas_Werkmeister_WMDE I completely understand why it's frustrating to not have more help in fixing this (I have been in this position myself with Selenium tests I wrote for GrowthExperiments), but can you please also try to see it from the perspective of people who are seeing failed builds on things that have nothing to do with Wikibase? I can say for myself, I just don't have the time to try to get into detail for why WDIO is behaving unpredictably here, and disabling the flaky test seems much less disruptive to the overall ecosystem of people who are relying on CI working in a timely and reliable way.

I’ve tried to think out loud, speculate what might be happening, wonder how WebdriverIO does or doesn’t behave, ask questions, all the while receiving zero responses from all the other people who ought to have a shared interest in making our tests less flaky

IMO the team here would be Quality-and-Test-Engineering-Team and I have just added them to this task earlier today. I don't think the engineers who have found this task through failed tests on their builds necessarily have the time or interest to try to address the underlying issues with the failed test.

It is very frustrating to then be accused of not doing enough about this flaky test.

FWIW, I see and very much appreciate your efforts to fix the underlying issue and improve the overall infrastructure.

my hope is that upgrading wdio-wikibase (and thus wdio-mediawiki) will help with the flakiness

Would it be acceptable to disable the test until this is done, and then try re-enabling again?

Would it be acceptable to disable the test until this is done, and then try re-enabling again?

+1. It has been my experience that flaky selenium tests are skipped if they remain flaky for too long. I'm not sure I agree with the revert of the patch that skipped the tests

For example, in CheckUser we had several selenium tests skipped on code that I had been trying to fix for months. However, there had been enough times that other developers had their CI jobs fail on unrelated extensions for the time pressure to dictate we skip until we had the time to fix them

As a compromise solution, is there a way for the selenium test to be skipped everywhere but in Wikibase? The issue I see here the test failures are causing issues in extensions that do not have any dependency on Wikibase, so as such if we can skip them there then there won't be the same time pressure to fix them but the test would still block merges in at least Wikibase extensions

Change #1196491 abandoned by Lucas Werkmeister (WMDE):

[mediawiki/extensions/Wikibase@master] Update wdio-mediawiki (5.1) and wdio-wikibase (6.1)

Reason:

it’s not that easy; see T406844 for details, and Ief5eeb2380 for the first next step

https://gerrit.wikimedia.org/r/1196491

Hey folks! I'm the Engineering Manager of the Quality Services team (formerly part of the QTE team). Kosta let me know about this ongoing discussion and I wanted to see if I could help out.

First off - I understand it's frustrating, working in isolation on intermittent test failures is the hardest part of test automation in my experience - because test automation is deceivingly simple. I appreciate all the work put in to try resolve the issue, and I've asked a member of my team to take a look at the test over the next few weeks and see if we can offer more detailed assistance as we try to get better at addressing flaky test automation ourselves.

As a specific piece recommendation: you can skip tests programmatically via Mocha, which may be an option depending on what environment variables or other markers are surfaced at runtime. There may be more graceful ways of doing this within the ecosystem of the CI platform too if you're using selenium specs, which could be worth looking into.

Based on the above discussion, I recommend skipping this test temporarily. My second recommendation would be to keep this ticket open or open a secondary ticket documenting that the test has been skipped and why. That way this can be treated as technical debt and gives my team and our Test Platform team time to look into the underlying cause of the instability and determine if there's a more robust fix available.

Finally, I'd like to thank people for the diligence - as a QA person it is gratifying to see that tests are valued here. As a longer-term action I'm working internally to WMF to establish a more robust way of handling test flakiness + other automated test concerns. Conversations like these help me understand the situation more fully so I can help make things better in the future.

I was part of doing the upgrade to webdriver.io 9 in core and saw errors like Failed to wait for mediawiki.base to be ready after 5000 ms. I was thinking this was caused by a change in webdriver.io but see now that i also happens in 7. I could also see that changing maxInstances: 4 to 1 helped in some cases.

I get a feeling this happens when we have more load on the CI but haven't been able to prove it. Been trying to minimize what Chromium do and I think more work can be done there. In general for all those flakiness problems work like T407947 would make it much easier for us to pinpoint and get rid off the problem.

We're tackling T406844: wdio-wikibase depends on wdio-mediawiki v2 first.

In T388228#11284252, @kostajh wrote:

Quality-and-Test-Engineering-Team

The team does not exist since at least December 2024, according to mediawiki.

As far as I know, nobody is monitoring Quality-and-Test-Engineering-Team project (but I might be wrong).

Wikibase is now using wdio-mediawiki v5 since yesterday (and prior to that, v3 since 22 October, v4 since 6 November). I think I still saw the flaky test at some point after the v3 upgrade, but not since the v4 upgrade, though I didn’t keep track of it closely. Please report here if you still see the issue after yesterday’s upgrade. (We’re not actually on the latest wdio-mediawiki yet, but we are at least on the latest WebdriverIO AFAIK – wdio-mediawiki v6 has different breaking changes that aren’t related to the wdio version.)

In T388228#11415264, @Lucas_Werkmeister_WMDE wrote:

Wikibase is now using wdio-mediawiki v5 since yesterday (and prior to that, v3 since 22 October, v4 since 6 November). I think I still saw the flaky test at some point after the v3 upgrade, but not since the v4 upgrade, though I didn’t keep track of it closely. Please report here if you still see the issue after yesterday’s upgrade. (We’re not actually on the latest wdio-mediawiki yet, but we are at least on the latest WebdriverIO AFAIK – wdio-mediawiki v6 has different breaking changes that aren’t related to the wdio version.)

wdio-mediawiki v5 is the latest wdio (v9). wdio-mediawiki v6 is api changes (see changelog).

Another flaky test: editEntityDatatypes.cy.ts, seen on https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php83/779/console

In T388228#11423184, @kostajh wrote:

Another flaky test: editEntityDatatypes.cy.ts, seen on https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-selenium-php83/779/console

⇒ T411266; let’s keep this issue focused on WebdriverIO failures.