Page MenuHomePhabricator

Document how to set up the testing environment and run browser tests per-patch.
Closed, ResolvedPublic3 Estimated Story Points

Description

Instructions are available at;
https://www.mediawiki.org/wiki/Reading/Setting_up_a_browser_test_instance

Related T100293 (seems mostly done, only missing the documentation).

Also related T104558 (regarding the Labs-Vagrant environment, most of the set up should be the same documentation).

See: https://www.mediawiki.org/wiki/Reading/Setting_up_a_browser_test_bot


  • The steps for configuring one of our staging/test instances to automatically run browser tests should be clearly documented and
  • Another engineer has reproduced the test environment following the documentation.

After sign-off publicly email explaining the process to wikitech-l.

Event Timeline

Jhernandez raised the priority of this task from to High.
Jhernandez updated the task description. (Show Details)

Post-sign off task is to publish this to wikitech-l.

I've left a comment on the talk page. I'll be adding more as I progress in testing. Please answer my comments there.

I'm stuck in review because of T106442. Will continue reviewing when that task is resolved. I was able to resolve the issue by creating a new instance.

When I run sudo ./install.sh I get the following error:

Add your new public key to gerrit at:  https://gerrit.wikimedia.org/r/#/settings/ssh-keys


cat: .pub: No such file or directory
Please enter a project name (example: Gather)
MobileFrontend
Please enter the name of extensions this project depends on (example: MobileFrontend)

Please enter a test tag (optional. example: smoke)

./install.sh: 167: ./install.sh: cannot create /home/bmansurov/barrybot/run.sh: Directory nonexistent
chmod: cannot access ‘/home/bmansurov/barrybot/run.sh’: No such file or directory
chown: invalid group: ‘bmansurov:bmansurov’
Just created /home/bmansurov/barrybot/run.sh, please make any modifications needed.

@bmansurov what username are you sshing in as? I think you used the same username for the bot which I suspect is not the intended behaviour?

@Jdlrobson, i used 'bmansurov' as was suggested by the script.

Sounds like an issue with the script not the documentation - I don't think it's intended that you use your own username. I'd suggest filing T104842

From my perspective the docs are pretty complete. There's just a few remaining problems with script. +1 from me.

OK, I'll wait until the script is fixed so that I can verify the documentation steps.

I ran through this one more time and I'm pretty confident this is in a good place now.
Someone when signing this off should setup a bot. You can reuse a new account I've setup called frankiebot (irc/ e-mail me for the password) to setup an instance for testing Gather browser tests. Hopefully your experience will be seamless.

So there is : Reading/Setting_up_a_browser_test_instance

I'm guessing that @rmoen's repo automates the steps on Reading/Setting_up_a_browser_test_bot.

Can we link to Reading/Setting_up_a_browser_test_bot in Reading/Setting_up_a_browser_test_instance where the barry bot installer is explained?

Moving back to -1 since not all AC seem to be fulfilled.

  • Another engineer has reproduced the test environment following the documentation.

Ping @Jdlrobson @rmoen @bmansurov

I reproduced the test environment for http://http://mobile-browser-tests.wmflabs.org/ following the instructions.
Unfortunately there are still a few issues which make this non-seamless

Pasting commit 460ff998ad5d4591626bedf959014a0978810886 with (is good = False)..
Traceback (most recent call last):
  File "./barrybot.py", line 240, in <module>
    watch( args )
  File "./barrybot.py", line 217, in watch
    paste_url = get_paste_url(output)
  File "./barrybot.py", line 177, in get_paste_url
    return output.split(': ')[1].strip()

I ran into the bundle install issue too
sh -c sudo -p " Your user account isn't allowed to install to the system Rubygems. You can cancel this installation and run: bundle install --path vendor/bundle
:(

@Jdlrobson All issues have been addressed. Bundle install now runs for the test user.
Language support for the test page is now added. Git and Git-review configs are being properly setup as the test user. Also I added the arcanist config so pasting should work now.

I'm going through this too. I'll update this task once I'm done.

So I installed barry with sudo ./install.sh and answered a bunch of questions. Switched to the account that was created for me sudo su bahabarry and ran cd ~/barrybot && ./run.sh. This is what I get

Could not connect to gerrit.
Enter your gerrit username: bmansurov
Trying again with ssh://bmansurov@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git
<traceback object at 0x7f9cf531e560>
We don't know where your gerrit is. Please manually create a remote
named "gerrit" and try again.
Traceback (most recent call last):
  File "/usr/local/bin/git-review", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/git_review/cmd.py", line 1173, in main
    config['hostname'], config['port'], config['project'])
  File "/usr/local/lib/python2.7/dist-packages/git_review/cmd.py", line 593, in check_remote
    add_remote(scheme, hostname, port, project, remote)
  File "/usr/local/lib/python2.7/dist-packages/git_review/cmd.py", line 333, in add_remote
    raise Exception("Could not connect to gerrit at %s" % remote_url)
Exception: Could not connect to gerrit at ssh://bmansurov@gerrit.wikimedia.org:29418/mediawiki/extensions/MobileFrontend.git
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Installing ast 2.0.0
Installing parser 2.2.0.3
Installing astrolabe 1.3.0
Installing builder 3.2.2
Installing ffi 1.9.9 with native extensions
Installing childprocess 0.5.6
Installing chunky_png 1.3.4
Installing diff-lcs 1.2.5
Installing multi_json 1.11.1
Installing gherkin 2.12.2 with native extensions
Installing multi_test 0.1.2
Installing cucumber 1.3.20
Installing i18n 0.7.0
Installing faker 1.4.3
Installing yml_reader 0.5
Installing data_magic 0.21
Installing dimensions 1.2.0
Installing unf_ext 0.0.7.1 with native extensions
Installing unf 0.1.4
Installing domain_name 0.5.24
Installing multipart-post 2.0.0
Installing faraday 0.9.1
Installing http-cookie 1.0.2
Installing faraday-cookie_jar 0.0.6
Installing headless 1.0.2
Installing json 1.8.2 with native extensions
Installing parallel 0.7.1
Installing rdiscount 2.1.8 with native extensions
Installing rkelly-remix 0.0.7
Installing jsduck 5.3.4
Installing mediawiki_api 0.4.1
Installing page_navigation 0.9
Installing rubyzip 1.1.7
Installing websocket 1.2.2
Installing selenium-webdriver 2.46.2
Installing watir-webdriver 0.8.0
Installing page-object 1.1.0
Installing mime-types 2.6.1
Installing netrc 0.10.3
Installing rest-client 1.8.0
Installing rspec-expectations 2.99.2
Installing syntax 1.2.0
Installing thor 0.19.1
Installing mediawiki_selenium 1.4.0
Installing powerpack 0.1.0
Installing rainbow 2.0.0
Installing ruby-progressbar 1.7.1
Installing rubocop 0.29.1
Using bundler 1.10.6
Bundle complete! 4 Gemfile dependencies, 49 gems now installed.
Bundled gems are installed into /home/bahabarry/.gem.
Set key 'default' = "https://phabricator.wikimedia.org" in user config (was null).
./run.sh: line 14: ./barrybot.py: No such file or directory

Also, when I ran echo 'foo' |arc paste I got the following:

Usage Exception: This command requires arc to connect to a Phabricator install, but no
Phabricator installation is configured. To configure a Phabricator URI:

  - set a default location with `arc set-config default <uri>`; or
  - specify `--conduit-uri=uri` explicitly; or
  - run `arc` in a working copy with an '.arcconfig'.

What did I do wrong?

@rmoen ⬆️? Seems like @bmansurov is hitting issues with the installer script?

Moving back to -1.

@bmansurov, not sure. Didn't have this issue. Seems like the arc uri isn't being set.

@rmoen @bmansurov Folks schedule some time together and try to troubleshoot, and if there is something to improve in the docs from that time do so.

This task could be considered done since Another engineer has reproduced the test environment following the documentation was done by @Jdlrobson already, but it would be great if we can get @bmansurov setup to work.

In any case, let's move this card along, or it's going to stay here in limbo too long.

@rmoen, when is a good time to go over this?

Blocked as it seems creating an instance is off the table.

no-instance.png (345×628 px, 16 KB)

What do we need to move this along?

Let's consider this done, pair up if you want to dig into this when you want :)