Page MenuHomePhabricator

Create bot that automatically rebases and rebuilds patches to master
Closed, ResolvedPublic5 Estimated Story Points

Description

Problem

When patches are submitted to the Popups extension, as soon a patch that touches JS is merged to master the assets become outdated (see epic)

Proposed solution

  • We will create a bot that watches changes on Popups master branch
  • When a change is merged to master, any patches which do not have unmerged parents will be downloaded, rebased. Rebase conflicts will be fixed by npm run build and then the patch resubmitted if there are no additional conflicts.

Developer notes

  • There's an ssh gerrit streaming api that you can listen to to get instant notifications that code has changed. Zuul uses it.
  • Implementor should also research how IRC bots get their information

Concerns around pushing non-rebased code.

There is an option with git push called force with lease that says git push should fail if the local tracking branch does not have the same root ref as the remote branch. e.g. stops you from pushing. Git review also allows you to rebase before pushing

Additional reading

Event Timeline

Jdlrobson created this task.Jun 6 2017, 6:34 PM
pmiazga added a subscriber: pmiazga.Jun 6 2017, 6:37 PM
demon removed a subscriber: demon.Jun 6 2017, 6:38 PM
Jdlrobson set the point value for this task to 5.Jun 7 2017, 4:45 PM

@hashar @zeljkofilipin @greg do you see any problems with the proposed solution?

greg added a comment.Jun 14 2017, 6:01 PM

I'll bring it up in our next team weekly (next week).

Jdlrobson updated the task description. (Show Details)

Neat!

Can get real time notifications via:

ssh -p 29418 gerrit.wikimedia.org gerrit stream-events

Let me see what I can do..

POC up and running.

The script here: https://gist.github.com/jdlrobson/a2dba042579e85e5180e5106d9ecec1a

rebased: https://gerrit.wikimedia.org/r/#/c/359573/

I just need to work out how to make it feed events into the python select library and then this will be real time and automatic (see my gist comment). Any ideas? @bmansurov how is your python?

I haven't looked too deeply, but you may want to remove the watch job in favor of a cron job for easy maintenance.

I've worked it out. 5 minutes away from computer was all I needed :)

Boom.
https://www.mediawiki.org/wiki/Reading/Web/Pixie.py_rebase_tool

It's up and running on the trending labs instance (for the time being)
I'm interested to see how long it remains running for.

If you want to use it simply comment on the gerrit patchset with "rebase".

It's currently hardcoded to Popups but would be very easy to turn it on for other repos.

@Jdlrobson Nice, great job with the docs!

A couple of comments:

  • It would be good to run npm install before npm run build in case something has changed in package.json before L70
  • Should we flush the logs periodically or keep the last N lines? (N = 10.000 or something like that) Otherwise the disk space may fill up at some point.

Great idea on commenting immediately before trying the rebase for feedback.

Moving to -1 to chat about these

It would be good to run npm install before npm run build in case something has changed in package.json before L70

Yep. I did originally do this for all runs but it's super slow, so I decided the same, just didn't implement it. I want to make sure this is useful before coding it some more.

Should we flush the logs periodically or keep the last N lines? (N = 10.000 or something like that) Otherwise the disk space may fill up at some point.

Can we not use logrotate ?

Yep. I did originally do this for all runs but it's super slow, so I decided the same, just didn't implement it. I want to make sure this is useful before coding it some more.

Try installing npm@5 on the instance, it has issues but installing works well, and it is a lot faster than previous versions, so it may make it bearable.

Can we not use logrotate ?

Yeah sure! Just a reminder since it is something that has bitten me before.

I've updated the script to run npm install now.
The script went down and I'm having difficulty getting the script to log reliably so I don't know why it did.

the bot is also running on RelatedArticles. How do we feel about switching that extension over to webpack... ?

Jdlrobson reassigned this task from Jdlrobson to Jhernandez.Jun 27 2017, 5:08 PM

Will signoff soon.

Jhernandez closed this task as Resolved.Jun 29 2017, 11:04 AM

Tested it on a patch, seems to work well and fast. Let's see how it serves us, maybe we can revisit in a few weeks to see if we've been using it.

@Jdlrobson Why do both BarryTheBrowserTestBot & Frankiebot interact with the patch and not just one?

Git review was configured with the other bot's username by accident. It amused me so I kept it. Just a shame I didn't call them B1 and B2 and make them have a dialogue. (Context: Bananas in pyjamas TV show)

I would note that Frankiebot overwrites the topic of the original change. It'd be good if we could keep this as:

  • A topic helps associate a set of changes (as Gerrit's UI now no longer shows the chain); and, the direct consequence
  • All rebased changes are associated with one another via Gerrit's UI. Boo.