The ref-updated hook does that. jtolds/gerrit-webhooks has an example of converting it into a webhook.

As a developer I want to be able to do things

What "things"? We have zuul and jenkins which can trigger pretty much any action when a patch comes in.

See blocked task for broader discussion please: T100293

@yuvipanda tells me we can use https://github.com/wikimedia/labs-tools-gerrit-to-redis

It should be enabled on the labs instance.

So... I got a proof of concept working. It's in /srv/mediawiki/extensions/Gather/tests/browser/Barry.sh and the tests pass when run locally. Currently it is super dumb and you have to manually run it. It doesn't download latest MobileFrontend and other dependencies yet.

The review it's given so far for Gather[1] are completely legit - the browser tests are passing/failing on that particular patch.

[1] https://gerrit.wikimedia.org/r/#/q/reviewer:%22BarryTheBrowserTestBot+%253Cjdlrobson%252Bbarry%2540gmail.com%253E%22,n,z

So the script that actually runs the browser test is not in a generic useful form but it is:
https://gist.github.com/jdlrobson/32b607f8009e897ee80c

It uses the GerritCommandLine tool to do grabbing and reviewing https://github.com/jdlrobson/GerritCommandLine

Ideally if we can use labs-tools-gerrit-to-redis for identifying patches and then pulling them down we wouldn't need the GerritCommandLine tool since the code to submit a review is pretty trivial and captured in this function:
https://github.com/jdlrobson/GerritCommandLine/blob/master/gerrit.py#L277

So step 1 to getting the gerrit to redis type stuff (gerrit stream it is called) working is to create an account for your gerrit bot, and then I can give it the stream right. I've given your account this right already, so you can test it locally (run ssh -p 29418 gerrit.wikimedia.org gerrit stream-events). You just need to implement this as well for your bot, and process the JSON coming out of the ssh.

Done using ssh -p 29418 gerrit.wikimedia.org gerrit stream-events which seems like the easiest way to do this sort of thing:
https://github.com/jdlrobson/Barry-the-Browser-Test-Bot/commit/51862e2263ed052409ce2a205c8977881fd5d465

barrybot has permissions to use this, so we could enable this on MobileFrontend or Gather as a test run.
This has potential to be noisy however, as the bot will run on every single patch and currently there is no way to cancel existing reviews mid-review since the review is done synchronously.

We may want to experiment first by making a bot run something simpler such as unit test coverage.

This is great. Do you think we should explore how to make this async (somehting like [1] or [2])? We could keep track of running processes/threads and kill them when we detect a new patchset. It sounds like a waste of resources if the tests keep running even after a new patchset has been uploaded.

[1] https://docs.python.org/3/library/multiprocessing.html
[2] https://docs.python.org/3/library/threading.html#module-threading

We could... but whether our time may be better spent elsewhere imo

If there were a way to monitor the number of builds that are run against outdated patches, then we could better estimate whether or not it were worth the effort.

How is the result of this spike – the patch, presumably – affected by mw:Continuous_integration/Browser_tests?

@phuedx I think we should aim to lean more on this > https://www.mediawiki.org/wiki/Continuous_integration/Browser_tests < yes and push that as far as we possibly can for our browser test coverage going forward. We'll be in new territory though so we need to tread carefully. I would prefer we invest time their however instead of working more on this task and related issues, given that setting up an instance is more painful than it needs to be even with automation.

I personally think the bots will be useful going forward for other purposes that release engineering do not have the time to pursue or are of interest to our team only, for example code review test coverage, so it's good to keep in our back pocket.

I personally think the bots will be useful going forward for other purposes that release engineering do not have the time to pursue or are of interest to our team only, for example code review test coverage, so it's good to keep in our back pocket.

+1

We've learned a lot from this experiment, e.g. just how hard it is to set up a labs instance for MobileFrontend and Gather. Now that Release-Engineering-Team are moving forward, we'll have more bandwidth to look at solving that problem.