Page MenuHomePhabricator

Spike [3hrs]: Making Barry work in real time
Closed, ResolvedPublic

Description

As a developer I want to be able to do things every time a new patch comes into MobileFrontend using Gerrit rather than every 30 minutes. Please take a look at how Jenkins works and how the Gerrit Reviewer tool works for inspiration.

@dduvall has been experimenting around this using Jenkins and made some progress https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-mw-selenium/42/consoleFull

  • How do these tools do this?
  • Create a proof of concept that demonstrates how we can do this.

Once we have this component we can use it to check out code on our labs instance (T101063) and then send messages to Gerrit ( T101068 )

Event Timeline

Jdlrobson raised the priority of this task from to Needs Triage.
Jdlrobson updated the task description. (Show Details)
Tgr added a subscriber: Tgr.Jun 2 2015, 2:04 AM

The ref-updated hook does that. jtolds/gerrit-webhooks has an example of converting it into a webhook.

Legoktm added a subscriber: Legoktm.Jun 2 2015, 2:08 AM

As a developer I want to be able to do things

What "things"? We have zuul and jenkins which can trigger pretty much any action when a patch comes in.

See blocked task for broader discussion please: T100293

Jdlrobson renamed this task from Spike: When a patch is submitted to Gerrit trigger a script to Spike: When a patch is submitted to Gerrit trigger a script that runs browser tests.Jun 10 2015, 10:31 PM
Jdlrobson set Security to None.

@yuvipanda tells me we can use https://github.com/wikimedia/labs-tools-gerrit-to-redis

It should be enabled on the labs instance.

So... I got a proof of concept working. It's in /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh and the tests pass when run locally. Currently it is super dumb and you have to manually run it. It doesn't download latest MobileFrontend and other dependencies yet.

The review it's given so far for Gather[1] are completely legit - the browser tests are passing/failing on that particular patch.

[1] https://gerrit.wikimedia.org/r/#/q/reviewer:%22BarryTheBrowserTestBot+%253Cjdlrobson%252Bbarry%2540gmail.com%253E%22,n,z

Jdlrobson added a comment.EditedJun 18 2015, 6:01 PM

So the script that actually runs the browser test is not in a generic useful form but it is:
https://gist.github.com/jdlrobson/32b607f8009e897ee80c

It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine

Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function:
https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277

So step 1 to getting the gerrit to redis type stuff (gerrit stream it is called) working is to create an account for your gerrit bot, and then I can give it the stream right. I've given your account this right already, so you can test it locally (run ssh -p 29418 gerrit.wikimedia.org gerrit stream-events). You just need to implement this as well for your bot, and process the JSON coming out of the ssh.

Jhernandez triaged this task as High priority.Jul 2 2015, 11:33 AM
Jdlrobson renamed this task from Spike: When a patch is submitted to Gerrit trigger a script that runs browser tests to Spike [3hrs]: When a patch is submitted to Gerrit trigger a script that runs browser tests.Jul 6 2015, 8:58 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson added a subscriber: dduvall.

Merge complete. Move to ready for dev at will if happy.

Jdlrobson renamed this task from Spike [3hrs]: When a patch is submitted to Gerrit trigger a script that runs browser tests to Spike [3hrs]: Making Barry work in real time.Jul 20 2015, 4:27 PM
Jdlrobson claimed this task.
Jdlrobson updated the task description. (Show Details)

Done using ssh -p 29418 gerrit.wikimedia.org gerrit stream-events which seems like the easiest way to do this sort of thing:
https://github.com/jdlrobson/Barry-the-Browser-Test-Bot/commit/51862e2263ed052409ce2a205c8977881fd5d465

barrybot has permissions to use this, so we could enable this on MobileFrontend or Gather as a test run.
This has potential to be noisy however, as the bot will run on every single patch and currently there is no way to cancel existing reviews mid-review since the review is done synchronously.

We may want to experiment first by making a bot run something simpler such as unit test coverage.

This is great. Do you think we should explore how to make this async (somehting like [1] or [2])? We could keep track of running processes/threads and kill them when we detect a new patchset. It sounds like a waste of resources if the tests keep running even after a new patchset has been uploaded.

[1] https://docs.python.org/3/library/multiprocessing.html
[2] https://docs.python.org/3/library/threading.html#module-threading

We could... but whether our time may be better spent elsewhere imo

If there were a way to monitor the number of builds that are run against outdated patches, then we could better estimate whether or not it were worth the effort.

How is the result of this spike – the patch, presumably – affected by mw:Continuous_integration/Browser_tests?

@phuedx I think we should aim to lean more on this > https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests < yes and push that as far as we possibly can for our browser test coverage going forward. We'll be in new territory though so we need to tread carefully. I would prefer we invest time their however instead of working more on this task and related issues, given that setting up an instance is more painful than it needs to be even with automation.

I personally think the bots will be useful going forward for other purposes that release engineering do not have the time to pursue or are of interest to our team only, for example code review test coverage, so it's good to keep in our back pocket.

I personally think the bots will be useful going forward for other purposes that release engineering do not have the time to pursue or are of interest to our team only, for example code review test coverage, so it's good to keep in our back pocket.

+1

We've learned a lot from this experiment, e.g. just how hard it is to set up a labs instance for MobileFrontend and Gather. Now that Release-Engineering-Team are moving forward, we'll have more bandwidth to look at solving that problem.

Jhernandez closed this task as Resolved.Jul 31 2015, 11:38 AM

So this is kind of done.

Do you folks think we should devote some time to migrating the browser tests running to the JJB stuff?