In T280652#10271787, @zeljkofilipin wrote:

In T280652#10268980, @kostajh wrote:

if a test is not useful enough to be disabled for a year, it should be deleted

I don't agree with this. Teams are busy. If you delete the test, it means there's almost zero chance of someone being able to go back and fix a flaky test that otherwise provided useful value in preventing regressions. I would leave them in place as documentation. I don't think it harms anything to leave them around.

I just picked 1 year as a reasonable time without much thought. If you think longer time is more reasonable, I'm open to discussion. (I think that time should be much shorter. 3 or 6 months.) After how long would it be reasonable to conclude nobody cares about a test? 2 years? 3 years?

I'd leave it indefinitely.

I don't agree that commented out code or skipped tests should be in our codebase. If a reminder for a team is needed, a task should be created. If the test is deleted, the code is not deleted from this universe. It's still in version control. The task can point to the commit that deletes the code. It's easy to get the code back.

It's documentation of something that, at some point, an individual or team thought was important. If it's disabled, it's due to a failure of implementation, but that doesn't take away from the importance of the test. For Selenium specifically, it means at a certain point the effort of trying to debug our WDIO setup is not worth the value we'd get out of the test.

I think I never explicitly stated the problem this task is trying to solve. Every time we do a major framework improvement, it breaks existing code. (For a list of major framework improvements, see Blog Post: Refactoring WebdriverIO Tests From Sync to Async Mode.) We usually update all broken code. If a test is disabled, it doesn't run, it doesn't break when the upgrade happens, so we don't fix it. After a year or two, there will be one or two major framework updates. If the code is enabled after being disabled for a year or two, there's a high chance it will not work.

The alternative to not updating disabled tests is enabling them during the migration. We make sure they work with the current framework, then we disable them again. That makes sure the test can be enabled, but it also wastes our time on code that nobody might care about any more.

I see. Then deleting makes sense. But alternatively, perhaps we could invest the effort in making the disabled tests more resilient and less flaky?

Change #1089338 had a related patch set uploaded (by Jared Blumer; author: Jared Blumer):

[mediawiki/extensions/GrowthExperiments@master] Remove skipped tests in addImage.js and delete homepage.js

https://gerrit.wikimedia.org/r/1089338

Change #1081080 abandoned by Zfilipin:

[mediawiki/extensions/GrowthExperiments@master] selenium: Delete tests disabled more than a year ago

https://gerrit.wikimedia.org/r/1081080

Change #1084066 abandoned by Zfilipin:

[mediawiki/extensions/Wikibase@master] selenium: Delete a test disabled almost a year ago

https://gerrit.wikimedia.org/r/1084066

Change #1083857 abandoned by Zfilipin:

[mediawiki/extensions/AbuseFilter@master] selenium: Delete tests disabled six months ago

https://gerrit.wikimedia.org/r/1083857

Change #1083859 abandoned by Zfilipin:

[mediawiki/extensions/CampaignEvents@master] selenium: Delete tests disabled six months ago

https://gerrit.wikimedia.org/r/1083859

Change #1089338 abandoned by Zfilipin:

[mediawiki/extensions/GrowthExperiments@master] Remove skipped tests in addImage.js and delete homepage.js

Reason:

Abandoning since the test will be moved to cypress.

https://gerrit.wikimedia.org/r/1089338

Change #1083861 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: Delete tests disabled more than two years ago

https://gerrit.wikimedia.org/r/1083861

Change #1084068 merged by jenkins-bot:

[mediawiki/core@master] selenium: Delete a test disabled almost two years ago

https://gerrit.wikimedia.org/r/1084068

Change #1084065 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1084065

Change #1133412 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CheckUser@master] selenium: Document when, how and why the test is skipped

https://gerrit.wikimedia.org/r/1133412

Change #1133415 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CommunityRequests@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1133415

Change #1133412 merged by jenkins-bot:

[mediawiki/extensions/CheckUser@master] selenium: Document when, how and why the test is skipped

https://gerrit.wikimedia.org/r/1133412

Change #1133415 merged by jenkins-bot:

[mediawiki/extensions/CommunityRequests@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1133415

Change #1135030 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/AbuseFilter@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1135030

Change #1135033 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CampaignEvents@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1135033

Change #1135034 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Wikibase@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1135034

Change #1135030 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1135030

Change #1135033 abandoned by Daimona Eaytoy:

[mediawiki/extensions/CampaignEvents@master] selenium: Delete a test disabled more than a year ago

Reason:

Done in Ibb88968119e378837d05af24adca97dcc27eadbf

https://gerrit.wikimedia.org/r/1135033

Change #1135034 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1135034

Complete for this round. Stalling this out for next round.

@zeljkofilipin Will organize a discussion to find an automated solution to this in Sep-October

This task should be replaced by a dashboard that would automatically refresh.