It looks like this one does still occur, per some central tracking in T277205

But there are none recently to look at....

addshore@contint2001:~$ grep -l "✖ should appear and disappear correctly" /srv/jenkins/builds/*-selenium-*/*/log

Seen again on 20 November 2023, though without the 10000ms part:

------------------------------------------------------------------
[Chrome 90.0.4430.212 linux #0-4] Running: Chrome (v90.0.4430.212) on linux
[Chrome 90.0.4430.212 linux #0-4] Session ID: 54ade820-cd52-495c-816a-96498934d47f
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] » /repo/tests/selenium/specs/tainted-ref.js
[Chrome 90.0.4430.212 linux #0-4] the Tainted icon
[Chrome 90.0.4430.212 linux #0-4]    ✖ should appear and disappear correctly
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] 1 failing (13.9s)
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] 1) the Tainted icon should appear and disappear correctly
[Chrome 90.0.4430.212 linux #0-4] Tainted Icon should be visible only after changing a statement value
Expect $(`.wb-tr-tainted-icon`) to exist

Expected: "exist"
Received: "not exist"
[Chrome 90.0.4430.212 linux #0-4] Error: Tainted Icon should be visible only after changing a statement value
[Chrome 90.0.4430.212 linux #0-4] Expect $(`.wb-tr-tainted-icon`) to exist
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] Expected: "exist"
[Chrome 90.0.4430.212 linux #0-4] Received: "not exist"
[Chrome 90.0.4430.212 linux #0-4]     at Context.<anonymous> (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/tainted-ref.js:47:43)
[Chrome 90.0.4430.212 linux #0-4]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
------------------------------------------------------------------
[Chrome 90.0.4430.212 linux #0-4] Running: Chrome (v90.0.4430.212) on linux
[Chrome 90.0.4430.212 linux #0-4] Session ID: 4be6370e-50ce-4fa3-99c4-b1d813382ccd
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] » /repo/tests/selenium/specs/tainted-ref.js
[Chrome 90.0.4430.212 linux #0-4] the Tainted icon
[Chrome 90.0.4430.212 linux #0-4]    ✖ should appear and disappear correctly
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] 1 failing (14.1s)
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] 1) the Tainted icon should appear and disappear correctly
[Chrome 90.0.4430.212 linux #0-4] Tainted Icon should be visible only after changing a statement value
Expect $(`.wb-tr-tainted-icon`) to exist

Expected: "exist"
Received: "not exist"
[Chrome 90.0.4430.212 linux #0-4] Error: Tainted Icon should be visible only after changing a statement value
[Chrome 90.0.4430.212 linux #0-4] Expect $(`.wb-tr-tainted-icon`) to exist
[Chrome 90.0.4430.212 linux #0-4]
[Chrome 90.0.4430.212 linux #0-4] Expected: "exist"
[Chrome 90.0.4430.212 linux #0-4] Received: "not exist"
[Chrome 90.0.4430.212 linux #0-4]     at Context.<anonymous> (/workspace/src/extensions/Wikibase/repo/tests/selenium/specs/tainted-ref.js:47:43)
[Chrome 90.0.4430.212 linux #0-4]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
------------------------------------------------------------------

I see this often, most recent failure: https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/15819/console

This is rather disruptive. I'm going to disable these tests. Someone should have done that like a year ago.

Change 983473 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/Wikibase@master] Disable frequently failing browser test

https://gerrit.wikimedia.org/r/983473

Change 983473 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Disable frequently failing browser test

https://gerrit.wikimedia.org/r/983473

In T266706#9410143, @gerritbot wrote:

Change 983473 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Disable frequently failing browser test

https://gerrit.wikimedia.org/r/983473

Absolutely unacceptable self+2 in my opinion. This code is our responsibility; you don’t get to unilaterally decide, with zero time for us to respond, that the feature being tested is actually irrelevant and it doesn’t matter if it gets broken. Please don’t do this again.

It would be helpful to have a policy (or maybe there is one already?) about flaky tests. My understanding is that for more than one case of a non-deterministic failure of a test, it is reasonable to disable it alongside a phabricator task explaining the error, so that there is an audit trail and relevant teams could follow up on it later.

@Lucas_Werkmeister_WMDE I'm sorry. I thought that this was a mostly abandoned component, as this task about it failing tests has been open for more than a year, and I've been annoyed by it causing test failures on my work. I won't do that again.

In T266706#9412289, @matmarex wrote:

@Lucas_Werkmeister_WMDE I'm sorry. I thought that this was a mostly abandoned component, as this task about it failing tests has been open for more than a year, and I've been annoyed by it causing test failures on my work. I won't do that again.

Alright, thank you. Tainted Refs is somewhat in limbo at the moment, but the Wikidata Dev Team (Quality Tools "Sprint") team is still responsible for it as far as I’m aware. (But I guess it shouldn’t go directly into their sprint board, so tagging all of Wikidata Dev Team for now.)

In T266706#9412278, @kostajh wrote:

It would be helpful to have a policy (or maybe there is one already?) about flaky tests. My understanding is that for more than one case of a non-deterministic failure of a test, it is reasonable to disable it alongside a phabricator task explaining the error, so that there is an audit trail and relevant teams could follow up on it later.

The closest thing I’m aware of is the etiquette that tasks which consistently break gate-and-submit for more than one Wikimedia-deployed repository are automatically eligible for Unbreak Now! status. (But I’m not sure if that’s actually written down somewhere, so I hesitate to call it a “policy”.) I’m not aware of any hard-and-fast rule on flaky tests; if a task has been open for long enough without activity from the responsible team, I think it would be acceptable for someone else to upload a change disabling that test, and for a third developer to +2 that if there are no objections after a while. (But I would still expect that +2 to come from a separate person, even if some time had passed.)