Change Details

We haven't defined a SLA or documented how we as a team should act on problems/requests with the tools we provide. Let us do that so that we and other teams know.

We haven't defined a SLA or documented how we as a team should act on problems/requests with the tools we provide. Let us do that so that we and other teams know. Let me explain better per tool: * **Selenium/webdriver.io/automated testing**: We need to define what we the #test-platform provides (see T374266), our testing principles and our strategy, And also now when we are a team providing this functionality I think we should take a more holistic approach to test automation and support more frameworks? For me personally I like the idea of having one recommended framework but also support others, in the way of you can choose your own testing framework as long as it provide metrics to our time series database so we can keep an eye on passed/test failures. I think we should provide a way to run those test and collect the metrics. Should we also help people to write "good" tests? Maybe in the framework we provide but I also think we can have recommendations that is independent on how to write good browser tests (for example, be defensive, write your test so it can take care of changes in the page, log failures etc). * **Pixel**: we need to provide documentation on what kind of service we provide. I think one valuable lesson from the current version is that it's unstable on how it depends on MediaWiki. It breaks and stops to work a lot and almost every time this is because of changes in extensions/setup. In the next iteration of pixel we should make sure that one test do not make other tests fail/not to run. And also make sure we have move the responsibility of when an extension breaks back to the developers instead of pixel. * **Synthetic performance test**: I think performance tests has a good first version of documentation on responsibilities/strategy and principles: https://wikitech.wikimedia.org/wiki/Performance/Synthetic_testing - feel free to give feedback what's missing or what should be changed. * **Quickstart**: I think we are missing the long term goal, strategies and make page at https://wikitech.wikimedia.org/ (when we have done the renaming). * **Catalyst and Sonar** : I think @SDunlap and @pwangai have more insights here. Then overall we need to define the structure for requests, how we handle alerts, and how we prioritise work on the different tools. Please feel free to add other things that I missed that is need to define/setup for our team :)

We haven't defined a SLA or documented how we as a team should act on problems/requests with the tools we provide. Let us do that so that we and other teams know. Let me explain better per tool: * **Selenium/webdriver.io/automated testing**: We need to define what we the #test-platform provides (see T374266), our testing principles and our strategy, And also now when we are a team providing this functionality I think we should take a more holistic approach to test automation and support more frameworks? For me personally I like the idea of having one recommended framework but also support others, in the way of you can choose your own testing framework as long as it provide metrics to our time series database so we can keep an eye on passed/test failures. I think we should provide a way to run those test and collect the metrics. Should we also help people to write "good" tests? Maybe in the framework we provide but I also think we can have recommendations that is independent on how to write good browser tests (for example, be defensive, write your test so it can take care of changes in the page, log failures etc). * **Pixel**: we need to provide documentation on what kind of service we provide. I think one valuable lesson from the current version is that it's unstable on how it depends on MediaWiki. It breaks and stops to work a lot and almost every time this is because of changes in extensions/setup. In the next iteration of pixel we should make sure that one test do not make other tests fail/not to run. And also make sure we have move the responsibility of when an extension breaks back to the developers instead of pixel. * **Synthetic performance test**: I think performance tests has a good first version of documentation on responsibilities/strategy and principles: https://wikitech.wikimedia.org/wiki/Performance/Synthetic_testing - feel free to give feedback what's missing or what should be changed. * **Quickstart**: I think we are missing the long term goal, strategies and make page at https://wikitech.wikimedia.org/ (when we have done the renaming). * **Catalyst and Sonar** : I think @SDunlap and @pwangai have more insights here. Then overall we need to define the structure for requests, how we handle alerts, and how we prioritise work on the different tools. Please feel free to add other things that I missed that is need to define/setup for our team :)