Page MenuHomePhabricator

[EPIC] Generate compiled assets from continuous integration
Open, Stalled, MediumPublic

Description

Background

mediawiki-extensions-Popups currently uses a compilation step (npm run build) to pull together front-end assets and process them before pointing ResourceLoader at them for serving.

Currently...

  • Compiled front-end assets are generated by developers and added to the git source tree under resources/dist on every patch that touches front-end sources.
  • We run a script on the npm job that verifies that the compiled assets are up to date, and bails if they weren't properly committed.

This allows us to:

  • Keep master updated with the deploy assets
  • Ensure that latest sources will have the latest deploy assets on master (with the CI step)

Cons

Rebase hell

Because of how it is set up, every pending patch on gerrit that touches the frontend assets, needs to be rebased every time master is updated, to get rid of the merge conflict and regenerate the compiled assets on top of it.

This can get very annoying for pending changes as any change merged will force a rebase on all pending changes.
It's so annoying we made a bot to do this for us (see T167181 Create bot that automatically rebases and rebuilds patches to master)

Example:

  1. There are changes A, B and C posted on gerrit based off master. They touch front-end assets and as such generate assets into resources/dist with npm run build
  2. Reviewer goes and +2s patch A.
  3. Goes to review B, wants to +2, but because A is merged to master and changed the resources/dist files, B now Cannot merge.
    • B and C now need to be rebased on top of master, run the build step, and re-submitted with updated compiled assets.

Cannot configure per wiki

When working on T173491 it was raised that because $wgScriptPath is configurable map files compiled from this process cannot be changed on a per wiki basis. A formal build step could potentially allow us to build on deploy and thus provide debugging on these wikis

What do we need

To not submit compiled assets with the patches, but keep the deploy branch (master currently) updated with the compiled assets.

Solutions considered

  1. Keep doing what we are doing
    • Pros: Works, nothing to change
    • Cons: Pretty painful if there is churn in the repo because of the merge conflicts
  2. Don't submit the compiled assets on every patch
    • Manually build and release to the master branch
    • Pros
      • No merge conflicts because of stale compiled assets
      • Control over the front-end source releases
    • Cons
      • Backend and front-end source could be out of sync on a release if assets not compiled appropriately
      • Front-end not auto updated in staging and beta cluster environments
      • Manual step, bound to fail because humans
  3. Make CI run the build step and merge compiled assets
    • With the pros of what we are doing currently, but without the cons
    • Pros
      • Simpler workflow compared to the current one as there's no need to remember to add the compiled assets to the change.
      • Keeps master updated with the deploy assets
      • Ensure that latest sources will have the latest deploy assets on deploy branch (master)
    • Cons
      • Needs work with RelEng team to help us automate this
        • Seems to be a dirty solution, having a separate deploy branch/repo seems to be preferred
      • There are jobs that need change to learn how to build assets, like the selenium jobs, otherwise their results will be moot

Ideally after chatting, option 3. Make CI run the build step and merge compiled assets would be our preferred choice if RelEng deems it feasible and not problematic. We had an initial chat with @hashar about different ways this could be accomplished.

Questions

  • Is this feasible? (amount of work to get it set up)
  • How often do we run the job/script to generate assets and merge to the deploy branch (master)? (hourly? daily? after each commit?)
  • Other repositories under wmf keep the deployable version of the project under a project/deploy branch. Is it worth it to have a separate deploy branch and sync it/submit compiled assets there? (Given this is a mediawiki extension) Or is it better to keep master as the deployable branch and push the assets there?
  • Is this a good chance to define a general CI process to trigger a build step on projects? (for bundling frontend assets, pre-compiling templates, optimizing static assets like images, etc)

Details

Related Gerrit Patches:
mediawiki/extensions/Popups : masterExperiment: Build assets via MediaWiki hook

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@hashar Obviously I've forgotten which tag I needed to add for release-engineering :( Was it Release-Engineering-Team ?

It's #together. It's not a joke. Try it out. ;)

Nice alias 😄

@hashar and @zeljkofilipin do you have any bandwidth over the next 2 weeks to help us with this?

Besides selenium/webdriverio I have little experience with npm jenkins jobs. I think this is a job for @hashar.

+ @demon for the release/build phase of the Popups extension.

I am going to first derail the topic before providing an actual answer.

Context // Antoine is ranting

T156333 proposes to use webpack to bundle all the javascript in a single file, and thus extension.json just have:

{
    "ResourceModules": {
        "ext.popups": {
            "scripts": [
                "resources/dist/index.js"
            ],
...

The inspiration came apparently from https://gerrit.wikimedia.org/r/#/c/333320/2/extension.json which show up one has to manually update the list of scripts being part of the module. My rant is why add an extra layer of complexity that nobody uses in MediaWiki community? Would it be sufficient to add globing support to extension registry? Eg:

{
    "ResourceModules": {
        "ext.popups": {
            "scripts": [
                "resources/ext.popups/**/*.js"
           ],

I don't really have an opinion but it would be rather nice to raise that little RFC out of just Popups. If it is beneficial for you, surely the whole MediaWiki community could well adopt it. Specially if that simplifies development.

Green grass all around

We have several source repositories that need an extra build step before we can deploy it. A few that comes to mind:

  • Wikidata which is a bundle of Wikibase extension + others
  • The npm based services which have a source repo defining dependencies in package.json, developers snapshot the installed node modules and push that in an intermediary repo eg: mediawiki/services/parsoid/deploy)
  • mediawiki/vendor.git which snapshot all composer dependencies we need.

Wikibase is done via a build script maintained by WMDE. Releng just grab whatever version ends up being tagged. Other cases are all handled manually.

Feedback

In both Wikidata and Parsoid cases, the canonical source repositories do not have the assets. An intermediary repo is used and we cut the deploy branch from that deployment repo.

I don't think we can easily support a Popups/deploy.git repository, there are just too much assumption everywhere that we except the extension to be in mediawiki/extensions/Popups.git . On the other hands, the Wikidata build process is known to work. Basically they do the build in that repo and tag/cut a branch then releng grabs that for deployment.

What can be investigated for Popups is:

  • Get rid of assets in Popups development branches requiring one to npm run build when developing
  • Cut deployment branches that would then have a commit adding the assets. The branch can be purged as they get undeployed, that should keep the repo at a sane size.
  • Have release engineering to stop automatically branching Popups@master and instead use the above deploy branch. An example for Wikidata https://gerrit.wikimedia.org/r/#/c/341489/1/make-wmf-branch/config.json (make-wmf-branch is the script we use to cut the wmf/* branches out of master, but supports alternatives to the master branch):
make-wmf-branch/config.json
"special_extensions": {
    "CentralNotice": "wmf_deploy",
    "Popups": "wmf_deploy",
    "Wikidata": "wmf/1.29.0-wmf.15",
}

CentralNotice use that because we want it deployed outside of the regular train in a specific deployment window (but maybe it will be made part of the train again and automatically cut on Tuesday like the rest).

With that system, on Tuesday releng will deploy whatever was the state of wmf_deploy branch. Or you can use semver branch to pin a version like Wikidata "wmf/1.29.0-wmf.15".

So most probably we can figure out a way to make that automatic. Probably a sample script at the root of Popups.git would be sufficient to create the branch, build asset and push for review.

Lastly there is the case of the beta cluster which just clones mediawiki/extensions.git master branch. Maybe we can hack the job to also run webpack when Popups is updated which apparently is just about: npm run build. Not sure what will break though =)

Unicorn world

Ideally the source repositories would have no asset and we would prepare all the assets via a magic build script. Which would take care of fetching and verifying the dependencies and assemble a huge pile of all the code we need. But we don't quite have anything of that kind :-/

I don't have much time for implementation currently (E_TOO_MANY_DUTIES). But can surely help / participate in refining this requirements.

Just to be more specific: ignore my rant about not using webpack. Lets focus on how we can:

  • get rid of those assets from the master branch
  • ship them in a repo/branch that is compatible with the existing assumptions
  • make it happen

And ideally propose such bundling system as the new MediaWiki standard ™.

Would it be sufficient to add globing support to extension registry?

The problem with this would be script order. Files need to be loaded in certain order and that where the complexity stems from. It is not a problem in small extensions, it is a big problem in big ones that leads to very obscure bugs. Some examples: T146031 T145566 T99461


The tooling is there for sound reasons, it allows us to consume and use libraries from npm, and it is going to allow us to pre-compile templates before sending them to the client, and other things.

@hashar As an extension having a separate branch for deployment seems quite controversial. I don't have any preferences though.

I'd be happy just having a job that builds daily and sends a patch for review to master.

We've chatted a bit on irc and there are jobs that require the compiled assets to be present, like the php ones that will complain if the script file on extension.json doesn't exist or the selenium jobs that actually run mediawiki to run the browser tests.

So either they learn how to build, or the built assets need to be there before running (git hooks?)

Would it be sufficient to add globing support to extension registry?

The tooling is there for sound reasons, it allows us to consume and use libraries from npm, and it is going to allow us to pre-compile templates before sending them to the client, and other things.

Yes I definitely understood. Consider my question a tongue-in-cheek / FUD :-} Sorry!


Another thing that came around. If we strip assets from the master branches, a bunch of CI jobs will fail. Typically the PHPUnit one run MediaWiki core structure tests that will complain because eg src/index.js is missing).

So we would need a way to define a build step and have CI to execute it. We don't usually don't run composer install on the CI jobs but rely instead on mediawiki/vendor.git. Though we have a script that inject MediaWiki core composer require-dev (eg phpunit). Maybe we can have an intermediary step to trigger a build, no idea how it would be defined, maybe extension.json.

Jhernandez updated the task description. (Show Details)Mar 9 2017, 4:17 PM
demon added a comment.Mar 9 2017, 6:56 PM

@hashar As an extension having a separate branch for deployment seems quite controversial. I don't have any preferences though.

Yes. This. Let's please not add more extensions needing custom branching logic--this is something we want to move away from, not encourage more of.

Is this a good chance to define a general CI process to trigger a build step on projects? (for bundling frontend assets, pre-compiling templates, optimizing static assets like images, etc)

+ 1. This does seem to be common and I'd love to free up more of release engineerings time by coming up with a generic solution. The ability to use npm test has been a huge time saver.

@hashar with regards to a MediaWiki standard, we are totally committed to that. Right now we are doing an experiment we feel strongly about with the view to share successes/failures as we discover them. Similar conversations are happening in other places via the frontend standards group.

@hashar with regards to a MediaWiki standard, we are totally committed to that. Right now we are doing an experiment we feel strongly about with the view to share successes/failures as we discover them. Similar conversations are happening in other places via the frontend standards group.

Yup that is 100% my understanding. And I know from experience that you like to generalize solutions to the whole ecosystem. I should have better written my large reply this afternoon in a more neutral sense instead of trying to add some humor tongue-in-cheek to it. Sorry about that!

hey Joaquin, we talked about this in the backlog grooming today. We'd like to get this scheduled for an upcoming sprint. Do you think you could coordinate with @hashar to find a time we could actually dedicate to doing this?

Jhernandez triaged this task as Medium priority.Apr 11 2017, 3:17 PM

We have a dependency on release engineering here. It would be great to coordinate to come up with a satisfactory solution instead of working on something just for ourselves.

Let's give some more time on the ticket for conversations.

Jdlrobson added a subscriber: greg.Apr 11 2017, 5:11 PM

Sure. My ask is that we talk to @greg and see if we can schedule some time in their busy schedule to work on this since this is causing us some problems.

Beside the quick chat early in March and the summary at T158980#3087905 , I do not plan to work on this nor do I have any idle cycles left to commit to it :-(

The main tension here seems to be rebasing at time of merge.

I think the following are probably the best solutions to this problem:

  1. Find some way to hook into the Gerrit rebase function and run npm run build at the end of it (not sure if this is possible)
  2. Bring back everyone's favourite browser bot and get him to watch changes and rebase them himself.
demon added a comment.May 26 2017, 7:59 PM
  1. Find some way to hook into the Gerrit rebase function and run npm run build at the end of it (not sure if this is possible)

This is not possible.

Jdlrobson renamed this task from Generate compiled assets from continuous integration to [EPIC] Generate compiled assets from continuous integration.Jun 2 2017, 8:31 PM
Jdlrobson added a project: Epic.
Jdlrobson raised the priority of this task from Medium to High.Jun 5 2017, 11:22 PM
greg moved this task from INBOX to Epics on the Release-Engineering-Team board.Jun 13 2017, 11:51 PM

For the record, I sent an email to wikitech-l asking how does a build process look like for extensions to gather feedback: https://lists.wikimedia.org/pipermail/wikitech-l/2017-June/088264.html

Jdlrobson updated the task description. (Show Details)Jun 21 2017, 11:59 PM
Jdlrobson added a subscriber: thcipriani.

Now we have a bot that helps merging how do people feel about compiling assets in other larger extensions..?

Seems like a topic for a different card.

Sure but what would that card look like? Are we ready to work on another extension? Which extension would it be?

Sent an email to discuss.

I replied to Reading-web-team] Using a bundler in another of our extensions
I suggest we revisit this in about a month.

demon removed a subscriber: demon.Jul 7 2017, 7:03 AM
phuedx added a comment.Jul 7 2017, 8:54 AM

I replied to Reading-web-team] Using a bundler in another of our extensions
I suggest we revisit this in about a month.

The Reading-Web-Team mailing list is a private forum. This is a public forum. Could you summarise what you said here?

greg added a comment.Aug 12 2017, 2:19 PM

I replied to Reading-web-team] Using a bundler in another of our extensions
I suggest we revisit this in about a month.

The Reading-Web-Team mailing list is a private forum. This is a public forum. Could you summarise what you said here?

Ping :)

Jdlrobson added a comment.EditedAug 15 2017, 3:30 PM

@greg sorry, I basically said: we're doing too many things at the task let's revisit this in a month (which is what I said in the comment)

To summarise the conversation that followed:

  • The build step is making the team's life a lot easier - easier to write, refactor and test code and opening up possibilities to innovate (in Popups we are now using ES6)
  • There were concerns MobileFrontend might not be the best extension to try this on next due to its size. RelatedArticles might be better.
  • The bot is a necessary evil. We don't want to rely on this, and are anxious about relying on it more, but we recognise to not rely on this, we need buy-in and support from release engineering to provide a build step and/or we need to skill up on understanding the integration config stack.

I think we're have more bandwidth to work on this later in the month/September.

Jhernandez removed Jhernandez as the assignee of this task.Aug 22 2017, 10:03 AM

We discussed this in passing a little today and mentioned that other repos such as the Wikipedia Portal and Wikidata have implemented solutions to this. Joaquin mentioned that these solutions, however, may not work well for MobileFrontend since it's expected to be treated as "just another extension" by CI.

Change 436327 had a related patch set uploaded (by Zfilipin; owner: Jdlrobson):
[mediawiki/extensions/Popups@master] Experiment: Build assets via MediaWiki hook

https://gerrit.wikimedia.org/r/436327

Change 436327 abandoned by Jdlrobson:
Experiment: Build assets via MediaWiki hook

Reason:
Trying https://gerrit.wikimedia.org/r/441515

https://gerrit.wikimedia.org/r/436327

Jdlrobson lowered the priority of this task from High to Low.Aug 30 2018, 1:30 AM
Jdlrobson raised the priority of this task from Low to Medium.Mar 8 2019, 8:04 PM
Niedzielski changed the task status from Open to Stalled.Mar 29 2019, 6:50 PM

I believe this is blocked on T199004.