I don't think that' this would be a good idea. Since we're not part of the gate and submit job anymore, it's even more important to have regular tests running so see, if somethings broken.

It's a bit bothersome, that ( apparently due to beta cluster being beta cluster ) tests randomly fail, but the last number in the Age column gives an idea if it's a random failure or not. - If this number is higher than one, it means, that the same test case failed two times in a row. - In the last weeks we never had that situation.

tl;dr: lets keep the daily suite running :-)

1. This doesn't look random any more. As said, I get these emails almost daily now.
2. What about the 17 minutes?

In T283139#7097582, @thiemowmde wrote:

1. This doesn't look random any more. As said, I get these emails almost daily now.

Yes, getting this email means, that at least one of the 47 test cases failed. In the report you get a list of the cases that failed. Only when the number in the age column is greater than one it would be a strong hint, that the test did not fail randomly. Otherwise it could be assumed it is random. Especially since the beta cluster can be flaky sometimes.

Not that it is a good thing to have randomly failing test cases on a regular basis, but there's already a ticket for that. T276082: Check randomly failing selenium daily tests

2. What about the 17 minutes?

I'm not sure why this is a issue. Sure, once a day we're binding resources to run that test suite, but I still think that it's important to have a regular check if things are well. And we also have this other ticket where we look fixing slow browser tests T282935: Clean-up slow TwoColConflict browser tests suite

2. What about the 17 minutes?

I'm not sure why this is a issue. Sure, once a day we're binding resources to run that test suite, but I still think that it's important to have a regular check if things are well. And we also have this other ticket where we look fixing slow browser tests T282935: Clean-up slow TwoColConflict browser tests suite

On thing to add here: The tests are also way slower, because they run on the beta cluster and things take more time there.

Here's a table of historical test failures, https://integration.wikimedia.org/ci/job/selenium-daily-beta-TwoColConflict/611/testReport/history/

As already stated, the results are randomly very flaky.

For comparison, here are the other daily tests: https://integration.wikimedia.org/ci/view/selenium-daily/

Only TwoColConflict and WikibaseLexeme tests take more than a few minutes, so there's certainly room for improvement. I don't have much of an opinion about leaving enabled or not, < 20 minutes once per day isn't a huge overhead relative to the gated tests, for example. But getting a failure email every other day doesn't bring much value, either. It's just a reminder that we own a complex and possibly flaky extension and that we need to stabilize the tests.

I've been working on support for parallel browser testing, we could experiment with that once the other low-hanging fruit is exhausted. Another quick fix would be to skip all but a handful of essential tests, maybe to cover what we consider the riskiest features.

The cleanup and optimizations done in the Two-Column-Edit-Conflict-Merge browser tests may relax the issue in that daily tests a bit more. Currently they seem to be way more stable.

It sill might make sense to have some threshold before sending out emails, but personally I wouldn't give it a very high priority.

Tests run more stable now. They also include a mechanism to repeat a failed test before it's marked as failed. As consequence the mails do not come in very often now and I would mark this ticket as resolved / invalid if you're fine with it @thiemowmde