Change 862310 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/MinervaNeue@master] selenium: Delete failing test

https://gerrit.wikimedia.org/r/862310

Change 862310 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] selenium: Delete failing test

https://gerrit.wikimedia.org/r/862310

Change 859505 abandoned by Zfilipin:

[mediawiki/extensions/WikibaseLexeme@master] selenium: Delete tests disabled for more than 1 year

Reason:

The tests seem to be fixed with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseLexeme/+/861355

https://gerrit.wikimedia.org/r/859505

Change 952002 had a related patch set uploaded (by Jforrester; author: Zfilipin):

[mediawiki/extensions/MobileFrontend@REL1_39] selenium: Delete all tests

https://gerrit.wikimedia.org/r/952002

Change 952002 merged by Jforrester:

[mediawiki/extensions/MobileFrontend@REL1_39] selenium: Delete all tests

https://gerrit.wikimedia.org/r/952002

Change 955803 had a related patch set uploaded (by Jforrester; author: Zfilipin):

[mediawiki/extensions/MobileFrontend@REL1_40] selenium: Delete all tests

https://gerrit.wikimedia.org/r/955803

Change 955803 abandoned by Jforrester:

[mediawiki/extensions/MobileFrontend@REL1_40] selenium: Delete all tests

Reason:

https://gerrit.wikimedia.org/r/955803

Change #1052733 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/AbuseFilter@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1052733

Change #1052738 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Echo@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1052738

Change #1053705 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CampaignEvents@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1053705

Change #1052738 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1052738

Change #1052733 merged by jenkins-bot:

[mediawiki/extensions/AbuseFilter@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1052733

Change #1053705 merged by jenkins-bot:

[mediawiki/extensions/CampaignEvents@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1053705

Change #1054075 had a related patch set uploaded (by Shyblumer; author: Shyblumer):

[mediawiki/skins/MinervaNeue@master] selenium: Add date and bug ID to skipped tests

https://gerrit.wikimedia.org/r/1054075

Change #1054343 had a related patch set uploaded (by Jbol; author: Jbol):

[mediawiki/extensions/CodeMirror@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1054343

Change #1054075 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] selenium: Add date and bug ID to skipped tests

https://gerrit.wikimedia.org/r/1054075

Change #1054343 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1054343

Change #1064793 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[integration/config@master] zuul: Add John Bolorinos to trusted users

https://gerrit.wikimedia.org/r/1064793

Change #1064793 merged by jenkins-bot:

[integration/config@master] Zuul: Add John Bolorinos to trusted users

https://gerrit.wikimedia.org/r/1064793

Change #1073210 had a related patch set uploaded (by Jbol; author: Jbol):

[mediawiki/extensions/VisualEditor@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1073210

Change #1073210 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1073210

Change #1077327 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/MinervaNeue@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077327

Change #1077328 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/GrowthExperiments@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077328

Change #1077331 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Wikibase@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077331

Change #1077334 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077334

Change #1077327 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077327

Change #1077328 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077328

Change #1077334 merged by jenkins-bot:

[mediawiki/core@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077334

Change #1077331 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1077331

Change #1078414 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/MinervaNeue@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1078414

Change #1078414 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] selenium: Document when, how and why a test is skipped

https://gerrit.wikimedia.org/r/1078414

Change #1080246 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CodeMirror@master] selenium: Delete 'CodeMirror bracket match highlighting for the wikitext 2017 editor'

https://gerrit.wikimedia.org/r/1080246

Change #1080597 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/skins/MinervaNeue@master] selenium: Delete tests disabled more than a year ago

https://gerrit.wikimedia.org/r/1080597

Change #1080597 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] selenium: Delete tests disabled more than a year ago

https://gerrit.wikimedia.org/r/1080597

Change #1081080 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/GrowthExperiments@master] selenium: Delete tests disabled more than a year ago

https://gerrit.wikimedia.org/r/1081080

Change #1083857 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/AbuseFilter@master] selenium: Delete tests disabled six months ago

https://gerrit.wikimedia.org/r/1083857

Change #1083859 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/CampaignEvents@master] selenium: Delete tests disabled six months ago

https://gerrit.wikimedia.org/r/1083859

Change #1083861 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Echo@master] selenium: Delete tests disabled more than two years ago

https://gerrit.wikimedia.org/r/1083861

if a test is not useful enough to be disabled for a year, it should be deleted

I don't agree with this. Teams are busy. If you delete the test, it means there's almost zero chance of someone being able to go back and fix a flaky test that otherwise provided useful value in preventing regressions. I would leave them in place as documentation. I don't think it harms anything to leave them around.

In T280652#10268980, @kostajh wrote:

if a test is not useful enough to be disabled for a year, it should be deleted

I don't agree with this. Teams are busy. If you delete the test, it means there's almost zero chance of someone being able to go back and fix a flaky test that otherwise provided useful value in preventing regressions. I would leave them in place as documentation. I don't think it harms anything to leave them around.

I just picked 1 year as a reasonable time without much thought. If you think longer time is more reasonable, I'm open to discussion. (I think that time should be much shorter. 3 or 6 months.) After how long would it be reasonable to conclude nobody cares about a test? 2 years? 3 years?

I don't agree that commented out code or skipped tests should be in our codebase. If a reminder for a team is needed, a task should be created. If the test is deleted, the code is not deleted from this universe. It's still in version control. The task can point to the commit that deletes the code. It's easy to get the code back.

I think I never explicitly stated the problem this task is trying to solve. Every time we do a major framework improvement, it breaks existing code. (For a list of major framework improvements, see Blog Post: Refactoring WebdriverIO Tests From Sync to Async Mode.) We usually update all broken code. If a test is disabled, it doesn't run, it doesn't break when the upgrade happens, so we don't fix it. After a year or two, there will be one or two major framework updates. If the code is enabled after being disabled for a year or two, there's a high chance it will not work.

The alternative to not updating disabled tests is enabling them during the migration. We make sure they work with the current framework, then we disable them again. That makes sure the test can be enabled, but it also wastes our time on code that nobody might care about any more.

Change #1084065 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/VisualEditor@master] selenium: Delete a test disabled more than a year ago

https://gerrit.wikimedia.org/r/1084065

Change #1084066 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/Wikibase@master] selenium: Delete a test disabled almost a year ago

https://gerrit.wikimedia.org/r/1084066

Change #1084068 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/core@master] selenium: Delete a test disabled almost two years ago

https://gerrit.wikimedia.org/r/1084068

Change #1080246 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@master] selenium: Delete 'CodeMirror bracket match highlighting for the wikitext 2017 editor'

https://gerrit.wikimedia.org/r/1080246

In T280652#10271787, @zeljkofilipin wrote:

In T280652#10268980, @kostajh wrote:

if a test is not useful enough to be disabled for a year, it should be deleted

I don't agree with this. Teams are busy. If you delete the test, it means there's almost zero chance of someone being able to go back and fix a flaky test that otherwise provided useful value in preventing regressions. I would leave them in place as documentation. I don't think it harms anything to leave them around.

I just picked 1 year as a reasonable time without much thought. If you think longer time is more reasonable, I'm open to discussion. (I think that time should be much shorter. 3 or 6 months.) After how long would it be reasonable to conclude nobody cares about a test? 2 years? 3 years?

I'd leave it indefinitely.

I don't agree that commented out code or skipped tests should be in our codebase. If a reminder for a team is needed, a task should be created. If the test is deleted, the code is not deleted from this universe. It's still in version control. The task can point to the commit that deletes the code. It's easy to get the code back.

It's documentation of something that, at some point, an individual or team thought was important. If it's disabled, it's due to a failure of implementation, but that doesn't take away from the importance of the test. For Selenium specifically, it means at a certain point the effort of trying to debug our WDIO setup is not worth the value we'd get out of the test.

I think I never explicitly stated the problem this task is trying to solve. Every time we do a major framework improvement, it breaks existing code. (For a list of major framework improvements, see Blog Post: Refactoring WebdriverIO Tests From Sync to Async Mode.) We usually update all broken code. If a test is disabled, it doesn't run, it doesn't break when the upgrade happens, so we don't fix it. After a year or two, there will be one or two major framework updates. If the code is enabled after being disabled for a year or two, there's a high chance it will not work.

The alternative to not updating disabled tests is enabling them during the migration. We make sure they work with the current framework, then we disable them again. That makes sure the test can be enabled, but it also wastes our time on code that nobody might care about any more.

I see. Then deleting makes sense. But alternatively, perhaps we could invest the effort in making the disabled tests more resilient and less flaky?

Change #1089338 had a related patch set uploaded (by Jared Blumer; author: Jared Blumer):

[mediawiki/extensions/GrowthExperiments@master] Remove skipped tests in addImage.js and delete homepage.js

https://gerrit.wikimedia.org/r/1089338

Change #1081080 abandoned by Zfilipin:

[mediawiki/extensions/GrowthExperiments@master] selenium: Delete tests disabled more than a year ago

https://gerrit.wikimedia.org/r/1081080

Change #1084066 abandoned by Zfilipin:

[mediawiki/extensions/Wikibase@master] selenium: Delete a test disabled almost a year ago

https://gerrit.wikimedia.org/r/1084066

Change #1083857 abandoned by Zfilipin:

[mediawiki/extensions/AbuseFilter@master] selenium: Delete tests disabled six months ago

https://gerrit.wikimedia.org/r/1083857

Change #1083859 abandoned by Zfilipin:

[mediawiki/extensions/CampaignEvents@master] selenium: Delete tests disabled six months ago

https://gerrit.wikimedia.org/r/1083859

Change #1089338 abandoned by Zfilipin:

[mediawiki/extensions/GrowthExperiments@master] Remove skipped tests in addImage.js and delete homepage.js

Reason:

Abandoning since the test will be moved to cypress.

https://gerrit.wikimedia.org/r/1089338

Change #1083861 merged by jenkins-bot:

[mediawiki/extensions/Echo@master] selenium: Delete tests disabled more than two years ago

https://gerrit.wikimedia.org/r/1083861