Page MenuHomePhabricator
Create Task
Maniphest T282840

Revisit how we run Selenium tests in our repos
Open, Needs TriagePublic
Actions

Description

I am frequently getting Wikibase Selenium failures on Vector. Usually these are due to race conditions and false positives. A recheck usually restores the problem but this wastes a lot of time in developer productivity and adds to CI traffic/slowdown.

As a consumer of these tests, there seems little value in running all browser tests on extensions and skins given we have a daily job that just does that. The only exception is for core changes, where I feel it is valuable to understand how a change there can impact extensions and skins.

Core: Run browser tests of all skins, core and extensions
Skins: Run browser tests of the skin.
Extensions: Run browser tests of the extension and all skins

Event Timeline

Jdlrobson created this task.May 13 2021, 10:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 13 2021, 10:52 PM
zeljkofilipin added a project: User-zeljkofilipin.May 14 2021, 10:34 AM
zeljkofilipin removed a project: Quality-and-Test-Engineering-Team.
zeljkofilipin added subscribers: hashar, zeljkofilipin.

@Jdlrobson Quality-and-Test-Engineering-Team is not in charge tests running in Continuous-Integration-Infrastructure. I think @hashar from Release-Engineering-Team is the one in charge of that.

zeljkofilipin added projects: Continuous-Integration-Config, Continuous-Integration-Infrastructure, Release-Engineering-Team.May 14 2021, 10:37 AM
zeljkofilipin moved this task from Backlog 🪒 to Watching 📺 on the User-zeljkofilipin board.
thcipriani edited projects, added Release-Engineering-Team (Radar); removed Release-Engineering-Team.May 18 2021, 5:39 PM
thcipriani subscribed.

The tests that we run for MediaWiki extensions and skins was @hashar's work 7 or 8 years ago; however, there is no one working on that currently. The workflow is implemented in Quibble so that's where to implement the change; however, developers should design what runs.

I believe this should be work shared between developers, platform, QTE, and our team providing instrumentation, but Release-Engineering-Team cannot make progress on this alone.

thcipriani moved this task from Limbo to Blocked on the Release-Engineering-Team (Radar) board.May 18 2021, 5:39 PM
Jdlrobson added a comment.May 18 2021, 6:07 PM

however, developers should design what runs.

Agreed, but my concern here is now what we run, but where we run the existing tests. Running all the tests for every repo delivers very little value IMO and slows down CI for everyone. I'd love to revisit this decision.

For example, as a Vector developer, I don't want Wikibase Selenium tests to run on every single patch there. The daily jobs are more than sufficient for catching those issues.

zeljkofilipin added a project: Browser-Tests.Oct 7 2021, 11:13 AM
zeljkofilipin moved this task from Watching 📺 to Backlog 🪒 on the User-zeljkofilipin board.Nov 14 2023, 3:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL