Page MenuHomePhabricator

Run browser tests on beta cluster from desktop domain
Closed, DuplicatePublic

Description

In MobileFrontend, we assume that the user always starts in the desktop view of the site.
We have an explicit step "I am using the mobile site" for asserting we are inside the mobile site.
As a result, tests that pass in @integration mode fail in this environment.

Let's update the environment variable so that it uses the correct domain for this job.
Note: There may be some old tests that are not tagged @integration that may need to be updated as a result of this change.

  • Update mediawiki_url in integration/config/jjb/browsertests.yaml
  • Update switch_views.feature to run in Jenkins on the beta cluster
  • Delete test job as part of sign off.

This work is currently blocked due to not having a decision on how this should work.
When we tried to do this previously we encountered issues with how the browser tests function - navigating to a URL uses an absolute URI as defined in browser_uri which always points to the desktop site. When a user switches to the mobile site future visits will not be redirected there.

Three potential solutions are:

  1. Specify mobile or desktop in URL e.g. visit(MainPage,true) where second parameter is isMobile
  2. Spoof user agent in all tests to force mobile site on all views
  3. Make switching to mobile set a cookie that forces a redirect from desktop to mobile on further visits

Details

Related Gerrit Patches:
mediawiki/extensions/MobileFrontend : masterRun switcher browser tests on beta cluster
mediawiki/extensions/MobileFrontend : masterRun browser test on beta cluster

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2016, 11:01 PM
Jhernandez triaged this task as Medium priority.May 25 2016, 4:42 PM

Changing to spike as I think the beta environment start with mobile now.

bmansurov renamed this task from Run browser tests on beta cluster from desktop domain to [Spike] [2 hours] Run browser tests on beta cluster from desktop domain.Jun 14 2016, 4:55 PM

@bmansurov seems actionable as is (i.e. not a spike)? What are you not sure about? Probably best to setup some time with @hashar or @zeljkofilipin to help merge patches, verify and do follow ups where needed.

Change 296418 had a related patch set uploaded (by Jdlrobson):
Run browser test on beta cluster

https://gerrit.wikimedia.org/r/296418

Jdlrobson renamed this task from [Spike] [2 hours] Run browser tests on beta cluster from desktop domain to Run browser tests on beta cluster from desktop domain.Jun 28 2016, 5:32 PM

Annoyingly I can't test these locally without having the password to the Selenium user account. Once someone has merged the above I can identify broken tests and promptly do a follow up to fix them (add the "I am on the mobile site" step)

The way we test such change is to create a copy of the existing Jenkins job, disable all notifications and then add a shell step that fetch and checkout the patchset.

The step by step:

A form would show up

  • Item Name: selenium-MobileFrontend-T130429
  • Copy existing Item: selenium-MobileFrontend

Save.

In the job configuration ( https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/configure ) delete the sections:

  • IRC Notification
  • Editable Notification

In the build section add in the first step Execute shell a command to fetch and checkout the patch:

mkdir -p "$WORKSPACE/log/junit"
git fetch https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend refs/changes/18/296418/2
git checkout -f FETCH_HEAD

Save and trigger a build of that selenium-MobileFrontend-T130429. It is going to fetch patchset 2 of https://gerrit.wikimedia.org/r/#/c/296418/ and run the browser tests :)

Proper credits go to Zeljko who has shown me and used that trick for ages!

Thanks Zeljko and @hashar - I've recorded this on wiki - https://www.mediawiki.org/wiki/Reading/Web/QA#Simulating_browser_test_run_on_an_unmerged_patchset. - feel free to edit / break out into another page with link!

bmansurov added a comment.EditedJun 29 2016, 1:56 AM

@bmansurov seems actionable as is (i.e. not a spike)? What are you not sure about?

I was confused by this comment (from the task description):

In MobileFrontend, we assume that the user always starts in the desktop view of the site.
We have an explicit step "I am using the mobile site" for asserting we are inside the mobile site.
As a result tests that pass in @integration mode fail in this environment.

This is not true as the tests use m. domain (thus spike - investigate whether the above quote is true at all) and your patch actually removes the mobile domain (which goes against the quote).

Apologies if my wording is confusing but the goal is to run mobile/desktop switching browser tests on a per commit basis. This is what the patch does and the test job shows it will not interfere with anything else.

The integration tests use the desktop domain. The beta cluster tests use the mobile domain (fixed by this patch).

Jhernandez added a subscriber: Jhernandez.

Needs some attention to comments on patch

Change 296418 merged by jenkins-bot:
Run browser test on beta cluster

https://gerrit.wikimedia.org/r/296418

You can see previously switcher tests were not run.

Now they do not seem to be running @hashar and @zeljkofilipin any ideas?

Fails with following error:

23:06:10 /usr/bin/ruby2.0 -S bundle exec cucumber tests/browser --backtrace --verbose --color --format pretty --format Cucumber::Formatter::Sauce --out '/mnt/jenkins-workspace/workspace/selenium-MobileFrontend/BROWSER/chrome/MEDIAWIKI_ENVIRONMENT/beta/PLATFORM/Linux/label/contintLabsSlave && UbuntuTrusty/log/junit' --tags ~@skip --tags @en.wikipedia.beta.wmflabs.org --tags @chrome
23:06:11 Code:
23:06:11   * tests/browser/features/support/env.rb
23:06:11 Failed to load 'jpg' programming language for file tests/browser/features/support/exif.jpg: cannot load such file -- cucumber/jpg_support/jpg_language
23:06:11   * tests/browser/features/support/exif.jpg [NOT SUPPORTED]
23:06:11   * tests/browser/features/support/hooks.rb
23:06:11   * tests/browser/features/support/pages/article_page.rb
23:06:11   * tests/browser/features/support/pages/create_article_page.rb
23:06:11   * tests/browser/features/support/pages/diff_page.rb
23:06:11   * tests/browser/features/support/pages/language_page.rb
23:06:11   * tests/browser/features/support/pages/main_page.rb
23:06:11   * tests/browser/features/support/pages/notification_page.rb
23:06:11   * tests/browser/features/support/pages/page.rb
23:06:11   * tests/browser/features/support/pages/special_history_page.rb
23:06:11   * tests/browser/features/support/pages/special_mobilediff_page.rb
23:06:11   * tests/browser/features/support/pages/special_search_page.rb
23:06:11   * tests/browser/features/support/pages/special_uploads_page.rb
23:06:11   * tests/browser/features/support/pages/special_userlogin_page.rb
23:06:11   * tests/browser/features/support/pages/user_page.rb
23:06:11   * tests/browser/features/support/pages/watchlist_page.rb
23:06:11 Failed to load 'sqlite' programming language for file tests/browser/features/support/permissions.sqlite: cannot load such file -- cucumber/sqlite_support/sqlite_language
23:06:11   * tests/browser/features/support/permissions.sqlite [NOT SUPPORTED]
23:06:11 Failed to load 'php' programming language for file tests/browser/LocalSettings.php: cannot load such file -- cucumber/php_support/php_language
23:06:11   * tests/browser/LocalSettings.php [NOT SUPPORTED]
23:06:11 Failed to load 'mediawiki' programming language for file tests/browser/README.mediawiki: cannot load such file -- cucumber/mediawiki_support/mediawiki_language
23:06:11   * tests/browser/README.mediawiki [NOT SUPPORTED]
23:06:11 Failed to load 'yml' programming language for file tests/browser/ci.yml: cannot load such file -- cucumber/yml_support/yml_language
23:06:11   * tests/browser/ci.yml [NOT SUPPORTED]
23:06:11   * tests/browser/environments.yml [NOT SUPPORTED]
23:06:11   * tests/browser/features/step_definitions/common_article_steps.rb
23:06:11   * tests/browser/features/step_definitions/common_steps.rb
23:06:11   * tests/browser/features/step_definitions/create_page_api_steps.rb
23:06:11   * tests/browser/features/step_definitions/diff_steps.rb
23:06:11   * tests/browser/features/step_definitions/editor_steps.rb
23:06:11   * tests/browser/features/step_definitions/editor_ve_steps.rb
23:06:11   * tests/browser/features/step_definitions/issues_steps.rb
23:06:11   * tests/browser/features/step_definitions/language_icon_steps.rb
23:06:11   * tests/browser/features/step_definitions/language_steps.rb
23:06:11   * tests/browser/features/step_definitions/mainmenu_steps.rb
23:06:11   * tests/browser/features/step_definitions/messages_steps.rb
23:06:11   * tests/browser/features/step_definitions/nearby_steps.rb
23:06:11   * tests/browser/features/step_definitions/notification_steps.rb
23:06:11   * tests/browser/features/step_definitions/pageactions_steps.rb
23:06:11   * tests/browser/features/step_definitions/references_steps.rb
23:06:11   * tests/browser/features/step_definitions/search_steps.rb
23:06:11   * tests/browser/features/step_definitions/special_contributions_steps.rb
23:06:11   * tests/browser/features/step_definitions/special_history_steps.rb
23:06:11   * tests/browser/features/step_definitions/special_watchlist_steps.rb
23:06:11   * tests/browser/features/step_definitions/switch_views.rb
23:06:11   * tests/browser/features/step_definitions/switch_views_bug_t129600.rb
23:06:11   * tests/browser/features/step_definitions/talk_steps.rb
23:06:11   * tests/browser/features/step_definitions/toc_steps.rb
23:06:11   * tests/browser/features/step_definitions/toggling_steps.rb
23:06:11   * tests/browser/features/step_definitions/ui_links_steps.rb
23:06:11   * tests/browser/features/step_definitions/user_page_steps.rb
23:06:11   * tests/browser/features/step_definitions/watchstar_steps.rb
23:06:11   * tests/browser/features/step_definitions/wikidata_descriptions.rb
23:06:11 
23:06:11 Features:
23:06:11 Parsing feature files took 0m0.088s
23:06:11 
23:06:11 0 scenarios
23:06:11 0 steps
23:06:11 0m0.000s
23:06:12 Recording test results
23:06:12 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
23:06:12 Performance: No threshold configured for making the test unstable
23:06:12 Performance: No threshold configured for making the test failure
23:06:12 
23:06:12 
23:06:12 
23:06:12 Performance: Recording JUnit reports 'log/junit/*.xml'
23:06:12 Archiving artifacts

The Failed to load ... are due to cucumber attempting to parse random non ruby files as specs. That can be ignored for now, fixed up later (maybe?).

The tests to run are marked with tags and cucumber is invoked with the filters:

--tags ~@skip --tags @en.wikipedia.beta.wmflabs.org --tags @chrome

So that is only going to run tests marked with @en.wikipedia.beta.wmflabs.org and @chrome and apparently none match:

0 scenarios
0 steps

The trick is the domain has been changed from .m. to desktop but the tag haven't been updated. Hence scenarios need to be added @en.wikipedia.beta.wmflabs whenever they have @en.m.wikipedia.beta.wmflabs.

Change 296703 had a related patch set uploaded (by Phuedx):
Update Cucumber scenario tags

https://gerrit.wikimedia.org/r/296703

phuedx claimed this task.Jun 30 2016, 10:45 AM
phuedx removed phuedx as the assignee of this task.Jun 30 2016, 5:19 PM

Un-licking this cookie /cc @Jdlrobson

Clicking the mobile view link doesn't force a redirect to the mobile domain, nor is it sticky, it's very difficult to assert "I am using the mobile site".

The next time Selenium navigates to a page, when it uses the desktop base domain, the user is bounced back to the desktop and thus all the tests fail. We don't have the same issue with @integration tests as these work on the same domain.

We could spoof the user agent to force the redirect via Varnish but my feeling is that all this work is not worth it for the sake of testing a link in the footer works.

What do others things? Should we decline?

Clicking the mobile view link doesn't force a redirect to the mobile domain, nor is it sticky, it's very difficult to assert "I am using the mobile site".
The next time Selenium navigates to a page, when it uses the desktop base domain, the user is bounced back to the desktop and thus all the tests fail. We don't have the same issue with @integration tests as these work on the same domain.
We could spoof the user agent to force the redirect via Varnish but my feeling is that all this work is not worth it for the sake of testing a link in the footer works.
What do others things? Should we decline?

I don't like it, but I agree declining might be the best option here.

Clicking the mobile view link doesn't force a redirect to the mobile domain, nor is it sticky, it's very difficult to assert "I am using the mobile site".

I wonder why? Why cannot we have the "I am using the mobile site" be run before each test? Why would selenium not continue from where left off? I'd like to find answers to these questions. Let's hold off on declining the task. If we have other priorities, I'd be happy to pick the task up later myself. This way I could learn more about our testing infrastructure too.

@bmansurov for caching reasons. In production the redirect to the mobile site is handled by varnish based on user agent. Without a mobile domain setup, the same URL will be used and MobileFrontend will decide based on the user agent. So if we want "I am using the mobile site" to work we would have to spoof the user agent or setup a mobile domain. Both are going to have their pain points to setup.

phuedx added a comment.Jul 6 2016, 8:16 AM

We could spoof the user agent to force the redirect via Varnish but my feeling is that all this work is not worth it for the sake of testing a link in the footer works.

I'd say that the link is important and that it exercises a fair amount of the MobileFrontend application.

Let's not decline this and investigate it further when the need arises (and we have time).

Annoyingly I can't test these locally without having the password to the Selenium user account. Once someone has merged the above I can identify broken tests and promptly do a follow up to fix them (add the "I am on the mobile site" step)

The password is documented here: https://office.wikimedia.org/wiki/Selenium_passwords

Change 296703 abandoned by Phuedx:
Run switcher browser tests on beta cluster

Reason:
Per T130429.

https://gerrit.wikimedia.org/r/296703

Jdlrobson updated the task description. (Show Details)

Okay. Updated description to describe the problem and moved.

Note that https://integration.wikimedia.org/ci/view/Mobile/job/Selenium-MobileFrontend-T130429/ is failing yet running on a daily basis. We should probably do something about that.