Change 948541 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/WikibaseLexeme@master] Actually disable running the daily selenium tests against beta

https://gerrit.wikimedia.org/r/948541

Change 948541 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Actually disable running the daily selenium tests against beta

https://gerrit.wikimedia.org/r/948541

Task Review

• This is essentially an "investigate and decide" task.
• If the decision's outcome is easy to implement, it should be implemented as part of this task.
• Easy to implement is understood as one line config change, for example.
• Before Picking this up or completing this task, a task to decide how to proceed with E2E tests in Lexeme should be created

Prio Notes

• Does not affect end users / production
• Affects monitoring
• Affects development efforts (Coming to a conclusion here could release us from having to "fix the tests")
• Does not affect onboarding efforts
• Might affect additional stakeholders (RelEng? Testing?)

My answers to the questions:

What were the original motivation for running these browser tests against beta?

Apparently this was / is a thing for many extensions, not just Lexeme: T188742: Run tests daily targeting beta cluster for all repositories with Selenium tests – @zeljkofilipin, can you say more? How much would you mind if we disabled these tests?

What benefits have we actually gained from doing that?

I don’t recall any bug that we noticed via these tests, and a quick look through the WikibaseLexeme git log didn’t show anything credited to beta either.

What costs do we incur from doing that?

They break quite frequently. Personally, I’ve stopped even paying attention to those emails, unless there’s a lot of them in a row.

In T344140#9160140, @Lucas_Werkmeister_WMDE wrote:

Apparently this was / is a thing for many extensions, not just Lexeme: T188742: Run tests daily targeting beta cluster for all repositories with Selenium tests – @zeljkofilipin, can you say more?

Selenium tests can target different environments, local and remote. See T344682 for example. Not all repositories work in all environments. Even if they do, they (might) behave different in different environments. (For example, an extension/skin can break another extension/skin). Some teams wanted to run their tests targeting the beta cluster.

In T344140#9160140, @Lucas_Werkmeister_WMDE wrote:

How much would you mind if we disabled these tests?

If this was a question for me: I don't care either way. If running tests targeting an environment (in this case the beta cluster) is useful for the team, do it. If it's not useful, don't do it.

What benefits have we actually gained from doing that?

From my (anecdotal) experience, the vast majority of failures are caused by the beta cluster being down.

Async story writing notes:

This is a technical decision that is needed. I think it would be nice to have a dedicated Decision Record in the WikibaseLexeme repo for future reference, rather just remembering this task here in Phabricator.
Since this should be a pretty reversible decision and independent of T333660, I'd be fine with that decision record being rather lightweight.

An additional argument or maybe option to consider beyond just "keep running the tests against beta daily" and "don't run the tests against beta daily" could be "run the tests daily in GitHub CI" like we added it for T333660: [ES-M5] Run EntitySchema CI at least once a day. But that could also be treated as a separate initiative.

So the ACs could be something like:

• create lightweight ADR for proposed decision
• make the decision
• can follow-up task for implementing the decision

I am not sure how this is an architectural decision, but have absolutely no objection recording this decision that way. I just wonder about whether details about the Wikimedia infrastructure should be recorded in a repository that is not necessarily only used in the Wikimedia context. In any case this is the CI being used for the codebase.

Regardless, I will add the AC. Thank you for the notes.

In T344140#9162887, @zeljkofilipin wrote:

In T344140#9160140, @Lucas_Werkmeister_WMDE wrote:

Apparently this was / is a thing for many extensions, not just Lexeme: T188742: Run tests daily targeting beta cluster for all repositories with Selenium tests – @zeljkofilipin, can you say more?

Selenium tests can target different environments, local and remote. See T344682 for example. Not all repositories work in all environments. Even if they do, they (might) behave different in different environments. (For example, an extension/skin can break another extension/skin). Some teams wanted to run their tests targeting the beta cluster.

In T344140#9160140, @Lucas_Werkmeister_WMDE wrote:

How much would you mind if we disabled these tests?

If this was a question for me: I don't care either way. If running tests targeting an environment (in this case the beta cluster) is useful for the team, do it. If it's not useful, don't do it.

Alright, thanks. The task looked like extensions were generally supposed to have these tests; if we can opt out of them, then I think we might do that.