All patches to MobileFrontend were blocked by an issue with CI - several browser tests which attempt to login to the mobile site failed either freezing on the login screen or not updating after a login. This indicated a problem with the login workflow. As a result we disabled these tests but now we need to restore them.
The problem is impacting both Minerva and MobileFrontend.
Here's an example of a failure:
https://integration.wikimedia.org/ci/job/mwselenium-quibble-docker/10791/consoleFull
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T139740 Port Selenium tests from Ruby to Node.js | |||
| Resolved | Jdlrobson | T162936 Improve developer happiness in reading web maintained extensions | |||
| Declined | None | T162256 [EPIC] Port Selenium tests from Ruby to Node.js on Reading Web extensions | |||
| Resolved | Jdforrester-WMF | T225031 On CI Jenkins, audit worker labels and remove unused ones | |||
| Open | None | T174018 Port Minerva's 7 Ruby-based browser tests to Selenium Node.js | |||
| Resolved | PRODUCTION ERROR | Jdlrobson | T219920 CI tests disabled: Steps that require login are disabled and should be restored. |
Change 500817 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Revert "MobileContext: Use correct method name setCookie() lieu of setcookie()"
I'm seeing occasional failures on https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/913/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/testReport/junit/(root)/Check%20UI%20components/Check_that_the_beta_mode_indicator_is_visible_in_beta_/ with a similar problem with login.
Change 500817 abandoned by Jdlrobson:
Revert "MobileContext: Use correct method name setCookie() lieu of setcookie()"
Reason:
Okay so this is likely a change in core/infrastructure.
The "I am logged into the mobile website" step seems to be defined inside the mediawiki selenium gem and that's what's failing here, so there's little i can do on the MobileFrontend level. @greg will need help from your team to debug this.
From what I can see the Ruby log_in method no longer works. Maybe there was an api/security fix?
Change 500860 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Disable browser tests on @integration with login steps
Change 500861 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] QA: Restore login steps
Change 500889 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] QA: Amend log in/using mobile site steps
Hey @zeljkofilipin I've found a way to get them passing https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/500861 (disabling them) so this is no longer broken and UBN, but I would appreciate help getting these enabled again and an explanation of what happened here, so that it doesn't happen again. I've lost a whole day of development activity due to this (and am spending my evening still troubleshooting) and I'd like to understand how we can avoid these problems in future.
Change 500860 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Disable browser tests on @integration with login steps
Change 501028 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Disable some essential tests because no other options
I'm disabling these tests as a short term measure, but that's not an acceptable long term solution.
So I've disabled quite a few tests (over 20) on Minerva and MobileFrontend so the builds pass and new patches are unblocked. The underlying issue seems to be the log_in step not working any more. It works fine for me when run locally but against the beta cluster I get " unexpected HTTP response (403) (MediawikiApi::HttpError)" when logging in. However, the integration tests are still ticking along (https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue/)
I'd like to get those tests back in a reasonable timeline as the logged in browser tests are pretty important.
https://gerrit.wikimedia.org/r/500861 is the patch I've been trying to enable these again with. I'd hope fixing the offending Ruby step in these would be relatively straightforward once we work out what changed.
That aside, I'm also open to using this problem to help prioritise porting the MobileFrontend/Minerva tests to Node.js which from my perspective is blocked on T199116. At minimum we need to get Ruby and Node.js browser tests running in both these repos side by side (T190710) and maybe this is a better use of our time in lieu of a quicker solution (even if it involves some kind of compromise on the how).
Zeljko are you free after the planning meeting tomorrow? If so maybe we could meet then.
Change 501028 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] QA: Disable some essential tests because no other options
Ruby framework has been deprecated since October 2017 (J79). I don't think it would be a good use of anybody's time to debug failing Ruby tests.
That aside, I'm also open to using this problem to help prioritise porting the MobileFrontend/Minerva tests to Node.js which from my perspective is blocked on T199116. At minimum we need to get Ruby and Node.js browser tests running in both these repos side by side (T190710) and maybe this is a better use of our time in lieu of a quicker solution (even if it involves some kind of compromise on the how).
Agreed. It might take me a few weeks to resolve T199116 (since I'm not really familiar with python/quibble/docker), but I think that is the blocker for a few tasks, not just this one.
@greg should I focus on T199116 for the next few weeks? I might need a bit of @hashar's help with that.
Change 501792 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Migrate editor_wikitext_saving.feature from Ruby to Node
Change 502349 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] QA: Port Notifications browser test to Node.js
Change 502593 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Restore category step for CI
Change 502615 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Setup the selenium daily job
Change 502615 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Setup the selenium daily job
Change 502593 abandoned by Jdlrobson:
Restore category step for CI
Reason:
not working as expected
Change 502659 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Disable broken Ruby browser tests
Change 502659 merged by Jdlrobson:
[mediawiki/extensions/MobileFrontend@master] Disable broken Ruby browser tests
Change 502822 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Beta mode cookie must be set on mobile domain
Change 502822 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Beta mode cookie must be set on mobile domain
Change 500861 abandoned by Jdlrobson:
QA: Restore login steps
Reason:
I've spent all this week fighting the Node.js framework and trying to get it to work for Minerva/MF. The Ruby framework seems to be unusable now and the mobile site is without a lot of the protection it's been enjoying. Any help appreciated getting these patches operational would be appreciated:
In particular the config patches:
https://gerrit.wikimedia.org/r/#/projects/integration/config,dashboards/default
Change 425422 had a related patch set uploaded (by Jdlrobson; owner: Jdrewniak):
[mediawiki/skins/MinervaNeue@master] Adding "diff.feature" selenium test
Change 502625 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Port remaining @login tests to Node.js
Change 425422 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Adding "diff.feature" selenium test
Change 501792 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Migrate editor_wikitext_saving.feature from Ruby to Node
Change 502349 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] QA: Port Notifications browser test to Node.js
Change 502625 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Port remaining @login tests to Node.js
From what I can see all browser tests are now restored.
A patch remains: https://gerrit.wikimedia.org/r/502593 however that's not essential for calling the restoring of these tests complete.
selenium-daily-beta-MobileFrontend is consistently failing. @Jdlrobson, should that block resolution of this task?
Yep. I'm getting twice daily failure emails as a reminder!
I think https://gerrit.wikimedia.org/r/506474 should get them green again. Fixing these is my priority today.
Change 502593 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Restore category step for CI