The MobileFrontend tests (in Firefox) have been flakey for some time. Take a look at the list on the left for examples of recurring issues.
I propose that when a test fails and the content of the Stacktrace contains one of
*Timeout::Error (Timeout::Error)
- Net::ReadTimeout
- 500 error (Our servers are currently experiencing a technical...)
the test should be marked as unreliable and should not be considered failure.
Could the test go amber when this happens instead? It would allow for better visual scanning when things break.
If there are 2 timeout errors but one actual error the test should stay red as there is a genuine error there.
Version: wmf-deployment
Severity: normal