Page MenuHomePhabricator

Create a Jenkins Job that builds the portal deployment artifacts in CI
Closed, ResolvedPublic

Description

Currently, the www.wikipedia.org and sister project portals are updated by running a build script on a developer machine, then a patch containing those artifacts are submitted to Gerrit for +2. The patch is then deployed during SWAT at semi-regular intervals.

Running the build locally and pushing it to Gerrit requires manual intervention, and instead, could be automated by Jenkins in CI.

We should create a Jenkins job that runs the build script (which can be combined into a single NPM command, like npm run-script build-all-portals) and creates the deployment artifacts. Jenkins could then create a commit from the artifacts and push it back to Gerrit for manual review and +2.

In summary, we need to build a Jenkins job that:

  1. Runs the portal build command at a regular interval (lets say, once a week)
  2. Creates a Gerrit patch with those artifacts
  3. Pushes the deployment artifacts to a dev server so a developer can visually review them (any static http server will do)
  4. Pushes the patch to Gerrit

Event Timeline

Jdrewniak created this task.Nov 3 2017, 2:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2017, 2:51 PM
Restricted Application added a project: Discovery. · View Herald TranscriptNov 3 2017, 2:52 PM
hashar added a comment.Nov 3 2017, 6:25 PM

We haven't reached to each other this week. Jan has set up a checkin for Monday 11/6.

Change 389468 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[integration/config@master] [WIP] JJB for building wikimedia portals

https://gerrit.wikimedia.org/r/389468

Change 389479 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] Add NPM script to run all portal build steps

https://gerrit.wikimedia.org/r/389479

Mentioned in SAL (#wikimedia-releng) [2017-11-07T15:15:00Z] <hashar> Created VPS account "PortalsBuilder" - T179694

Mentioned in SAL (#wikimedia-releng) [2017-11-07T16:15:02Z] <hashar> Created portalsbuilder in Gerrit, generated a ssh key pair for it and stored in Jenkins credentials store - T179694

debt triaged this task as Normal priority.Nov 7 2017, 4:34 PM
debt moved this task from Backlog to In Progress on the Discovery-Portal-Sprint board.
hashar added a comment.Nov 7 2017, 5:05 PM

Status

Jan and I had a chat yesterday about it.

https://gerrit.wikimedia.org/r/#/c/389479/ introduces an entry point build-all-portals to generate the assets. Jenkins will call it.

Jan and Hashar paired on the Jenkins job https://gerrit.wikimedia.org/r/389468 . It will:

  • clone the repos on Monday at 7am
  • run npm build-all-portals
  • craft a git commit and send it for review to Gerrit as user PortalsBuilder
  • report by email

https://integration.wikimedia.org/ci/job/wikimedia-portals-build/4/console

So eventually we had a change ending up in Gerrit:

Assets build - 2017-11-09 12:47:38+00:00
https://gerrit.wikimedia.org/r/#/c/390234/

:D

Change 389479 merged by jenkins-bot:
[wikimedia/portals@master] Add npm script to run all portal build steps

https://gerrit.wikimedia.org/r/389479

hashar added a comment.Nov 9 2017, 2:53 PM

Lets see how it behaves Monday.

I guess next we will want to look at pushing changes to a new /deploy repository and have operations/mediawiki-config use that.

Change 389468 merged by jenkins-bot:
[integration/config@master] Job for building wikimedia portals

https://gerrit.wikimedia.org/r/389468

@hashar when thinking about splitting the deployment assets into their own repo, I had some concerns regarding developer workflow with a git submodule. Specifically, it seemed confusing to me that the artifacts repo would be updated by the jenkins job, then mediawiki-config would pull to the head of that repo, but the source repo would be stuck at whatever commit the submodule was added at (unless a patch was submitted to update the submodule). Luckily, I discovered that git submodules can also track branches, https://stackoverflow.com/questions/9189575/git-submodule-tracking-latest/9189815#9189815 and that sounds like a pretty good option to me :)

Change 391189 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[integration/config@master] Changing time of wikimedia-portals-build job

https://gerrit.wikimedia.org/r/391189

Change 391189 merged by jenkins-bot:
[integration/config@master] Changing time of wikimedia-portals-build job

https://gerrit.wikimedia.org/r/391189

greg added a comment.Nov 14 2017, 11:52 PM

Mentioned in SAL (#wikimedia-releng) [2017-11-07T16:15:02Z] <hashar> Created portalsbuilder in Gerrit, generated a ssh key pair for it and stored in Jenkins credentials store - T179694

@hashar This gerrit user's email is our team internal list, so we're all getting gerrit update emails for each patch. We should do something about that. Can we use a different email/list for that?

greg added a comment.Nov 15 2017, 12:40 AM

Can we use a different email/list for that?

Really, the account should be owned by the Portals team, not RelEng, right?

greg added a comment.Nov 15 2017, 5:19 PM

Really, the account should be owned by the Portals team, not RelEng, right?

@Jdrewniak what would make sense from your side?

Really, the account should be owned by the Portals team, not RelEng, right?

@Jdrewniak what would make sense from your side?

Not sure. In reality, the portals team is just me and @debt, and we don't have an email/list for the portals. It probably shouldn't be a personal email, so I guess we can make a portals email list and use that instead?

greg added a comment.Nov 17 2017, 8:01 PM

Not sure. In reality, the portals team is just me and @debt, and we don't have an email/list for the portals. It probably shouldn't be a personal email, so I guess we can make a portals email list and use that instead?

+1

debt added a comment.Nov 17 2017, 10:02 PM

@Jdrewniak and @greg — I'll ask OIT to create an email list for this purpose. :)

There is now a portals@wikimedia.org however it is public so we cant really have a Wikitech account attached to it. Else one can trivially retrieve credentials by asking for a temporary password. Grrr :-(

I think what we did for other bots is to have a list that is an alias / relay to several emails. But not a proper list with archives etc. I am not quite sure how to set that up. Alternatively, the current list can be restricted / made private and that would solve the issue.

From there, we can:

  • change the email for the PortalsBuilder user on wikitech
  • adjust the git author/committer email in the job
  • update the email the jenkins job send spam too
debt added a comment.Nov 21 2017, 5:53 PM

There is now a portals@wikimedia.org however it is public so we cant really have a Wikitech account attached to it. Else one can trivially retrieve credentials by asking for a temporary password. Grrr :-(
I think what we did for other bots is to have a list that is an alias / relay to several emails. But not a proper list with archives etc. I am not quite sure how to set that up. Alternatively, the current list can be restricted / made private and that would solve the issue.
From there, we can:

  • change the email for the PortalsBuilder user on wikitech
  • adjust the git author/committer email in the job
  • update the email the jenkins job send spam too

Ahh....I didn't realize that we didn't want it public. I'll see what I can do to change the portals@wikimedia.org list.

Nice! I will change the registered email / jenkins job etc :)

debt added a comment.Nov 28 2017, 4:32 PM

Hi @hashar - did the email get setup? I don't think I received one yesterday after the build. Thanks for checking! :)

Hi @hashar - did the email get setup? I don't think I received one yesterday after the build. Thanks for checking! :)

Nevermind....! We found the email and it all seemed to work, yay!

debt closed this task as Resolved.Nov 28 2017, 4:55 PM
hashar reopened this task as Open.EditedDec 21 2017, 1:34 PM

From T179694#3778354 , I have missed updating the job to use the private list portals@lists.wikimedia.org (created via T180976)

  • change the email for the PortalsBuilder user on wikitech
  • adjust the git author/committer email in the job
  • update the email the jenkins job send spam too

Mentioned in SAL (#wikimedia-releng) [2017-12-21T13:51:47Z] <hashar> wikitech: change email of PortalsBuilder user from releng@lists.wikimedia.org to portals@lists.wikimedia.org | Credentials come from https://phabricator.wikimedia.org/D872 | T179694

Change 399630 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Update PortalsBuilder email

https://gerrit.wikimedia.org/r/399630

  • change the email for the PortalsBuilder user on wikitech
  • adjust the git author/committer email in the job
  • update the email the jenkins job send spam too

I have updated the PortalsBuilder password on wikitech. Which triggers:

A confirmation email has been sent to the specified email address. Before any other email is sent to the account, you will have to follow the instructions in the email, to confirm that the account is actually yours.

So @Jdrewniak / @debt would have received a mail from Wikitech to portals@lists.wikimedia.org asking for confirmation of the email.

I have logged in to Gerrit and confirmed ( https://gerrit.wikimedia.org/r/#/settings/ ) that the email address for the account is updated. One can check by searching in Gerrit for owner:portalsbuilder which would autocomplete to owner:"PortalsBuilder <portals@lists.wikimedia.org>"


The Jenkins job wikimedia-portals-build is updated via https://gerrit.wikimedia.org/r/399630

I have took on me to update the job and trigger a build https://integration.wikimedia.org/ci/job/wikimedia-portals-build/22/console

https://gerrit.wikimedia.org/r/#/c/399631/ the commit metadata has:

Author: PortalsBuilder <portals@lists.wikimedia.org>
Committer: PortalsBuilder <portals@lists.wikimedia.org>

So I guess it is all fine now!

Change 399630 merged by jenkins-bot:
[integration/config@master] Update PortalsBuilder email

https://gerrit.wikimedia.org/r/399630

Left to do:

  • validate the new email in wikitech based on the email sent to portals@lists.wikimedia.org
  • allow jenkins-bot@wikimedia.org to post to portals@lists.wikimedia.org

Jenkins send the email with jenkins-bot <jenkins-bot@wikimedia.org>. However the portals list is private and messages sent by non-members are held into moderation. So I guess that email should be made allowed to post or added as a member (with message delivery disabled).

jenkins-bot@wikimedia.org is an alias to release engineering if I remember properly.

debt added a comment.Dec 21 2017, 3:07 PM

Hi @hashar -- I've done the following:

jenkins-bot@wikimedia.org has been successfully subscribed to Portals.
gerrit@wikimedia.org has been successfully subscribed to Portals.

and I can confirm that we did get the emails send from the PortalsBuilder.

Let me know if there is anything else you'd like us to do! :)

Nice! I have rebuild the job once again ( build #23). That should have caused Gerrit to emit a few notifications to the list. If so I guess we can finally mark this resolved :]

debt added a comment.Dec 21 2017, 9:12 PM

Hmmm...I don't see that an email was posted to the list, but as long as it didn't bounce back to Gerrit, that's good? Maybe? Or, the list is setup not in a way that is truly useful?

I have commented on https://gerrit.wikimedia.org/r/#/c/399687 which should have sent an email to the list. Maybe the message is still hold into moderation? :(

greg added a comment.Dec 21 2017, 11:56 PM

Hmmm...I don't see that an email was posted to the list, but as long as it didn't bounce back to Gerrit, that's good? Maybe? Or, the list is setup not in a way that is truly useful?

I diagnosed this with @debt on a hangout. I think this is now resolved? Resolve this loooong task now? :) :)

hashar closed this task as Resolved.Dec 22 2017, 9:46 AM

Thank you very much @greg and @debt ! The job will trigger on Monday, if something is odd we can reopen this :]

debt added a comment.Jan 2 2018, 8:56 PM

Sweet, @hashar! I think we'll need to take a look at this next week to be sure nothing is off/odd about it, since January 1st wasn't a normal deploy day (due to it being New Years Day). :)