[EPIC] Generate compiled assets from continuous integration
Open, Stalled, MediumPublic
Actions

Assigned To

None

Authored By

	• Jhernandez
	Feb 24 2017, 6:20 PM

Description

Background

mediawiki-extensions-Popups currently uses a compilation step (npm run build) to pull together front-end assets and process them before pointing ResourceLoader at them for serving.

Currently...

Compiled front-end assets are generated by developers and added to the git source tree under [resources/dist](https://github.com/wikimedia/mediawiki-extensions-Popups/tree/master/resources/dist) on every patch that touches front-end sources.
We run a script on the npm job that verifies that the compiled assets are up to date, and bails if they weren't properly committed.

This allows us to:

Keep master updated with the deploy assets
Ensure that latest sources will have the latest deploy assets on master (with the CI step)

Cons

Rebase hell

Because of how it is set up, every pending patch on gerrit that touches the frontend assets, needs to be rebased every time master is updated, to get rid of the merge conflict and regenerate the compiled assets on top of it.

This can get very annoying for pending changes as any change merged will force a rebase on all pending changes.
It's so annoying we made a bot to do this for us (see T167181 Create bot that automatically rebases and rebuilds patches to master)

Example:

There are changes A, B and C posted on gerrit based off master. They touch front-end assets and as such generate assets into resources/dist with npm run build
Reviewer goes and +2s patch A.
Goes to review B, wants to +2, but because A is merged to master and changed the resources/dist files, B now Cannot merge.
- B and C now need to be rebased on top of master, run the build step, and re-submitted with updated compiled assets.

Cannot configure per wiki

When working on T173491 it was raised that because $wgScriptPath is configurable map files compiled from this process cannot be changed on a per wiki basis. A formal build step could potentially allow us to build on deploy and thus provide debugging on these wikis

What do we need

To not submit compiled assets with the patches, but keep the deploy branch (master currently) updated with the compiled assets.

Solutions considered

Keep doing what we are doing
- Pros: Works, nothing to change
- Cons: Pretty painful if there is churn in the repo because of the merge conflicts
Don't submit the compiled assets on every patch
- Manually build and release to the master branch
- Pros
  - No merge conflicts because of stale compiled assets
  - Control over the front-end source releases
- Cons
  - Backend and front-end source could be out of sync on a release if assets not compiled appropriately
  - Front-end not auto updated in staging and beta cluster environments
  - Manual step, bound to fail because humans
Make CI run the build step and merge compiled assets
- With the pros of what we are doing currently, but without the cons
- Pros
  - Simpler workflow compared to the current one as there's no need to remember to add the compiled assets to the change.
  - Keeps master updated with the deploy assets
  - Ensure that latest sources will have the latest deploy assets on deploy branch (master)
- Cons
  - Needs work with RelEng team to help us automate this
    - Seems to be a dirty solution, having a separate deploy branch/repo seems to be preferred
  - There are jobs that need change to learn how to build assets, like the selenium jobs, otherwise their results will be moot

Ideally after chatting, option 3. Make CI run the build step and merge compiled assets would be our preferred choice if RelEng deems it feasible and not problematic. We had an initial chat with @hashar about different ways this could be accomplished.

Questions

Is this feasible? (amount of work to get it set up)
How often do we run the job/script to generate assets and merge to the deploy branch (master)? (hourly? daily? after each commit?)
Other repositories under wmf keep the deployable version of the project under a project/deploy branch. Is it worth it to have a separate deploy branch and sync it/submit compiled assets there? (Given this is a mediawiki extension) Or is it better to keep master as the deployable branch and push the assets there?
Is this a good chance to define a general CI process to trigger a build step on projects? (for bundling frontend assets, pre-compiling templates, optimizing static assets like images, etc)

Details

	Subject	Repo	Branch	Lines +/-
	Experiment: Build assets via MediaWiki hook	mediawiki/extensions/Popups	master	+31 -5

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		Jdlrobson	T162936 Improve developer happiness in reading web maintained extensions
Stalled		None	T158980 [EPIC] Generate compiled assets from continuous integration
Resolved		• Jhernandez	T167181 Create bot that automatically rebases and rebuilds patches to master
Declined		None	T199004 RFC: Add a frontend build step to skins/extensions to our deploy process
Declined		None	T257061 Evaluate the workflows and workload of managing a Node package repository
Stalled		None	T257068 Draft: RFC: Evaluate alternative Node package managers for improved package security
Open		None	T257072 Determine Node package auditing workflows
Open		None	T315929 Reconsider build step in Popups and MobileFrontend
Resolved	BUG REPORT	ovasileva	T329901 [unplanned] Officially drop ES5 support for page previews
Open		None	T379793 [EPIC] Remove Webpack from MobileFrontend
Resolved		Jdlrobson	T379822 Stop using webpack in mobile.editor.overlay module

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Is this a good chance to define a general CI process to trigger a build step on projects? (for bundling frontend assets, pre-compiling templates, optimizing static assets like images, etc)

+ 1. This does seem to be common and I'd love to free up more of release engineerings time by coming up with a generic solution. The ability to use npm test has been a huge time saver.

@hashar with regards to a MediaWiki standard, we are totally committed to that. Right now we are doing an experiment we feel strongly about with the view to share successes/failures as we discover them. Similar conversations are happening in other places via the frontend standards group.

@hashar with regards to a MediaWiki standard, we are totally committed to that. Right now we are doing an experiment we feel strongly about with the view to share successes/failures as we discover them. Similar conversations are happening in other places via the frontend standards group.

Yup that is 100% my understanding. And I know from experience that you like to generalize solutions to the whole ecosystem. I should have better written my large reply this afternoon in a more neutral sense instead of trying to add some humor tongue-in-cheek to it. Sorry about that!

hey Joaquin, we talked about this in the backlog grooming today. We'd like to get this scheduled for an upcoming sprint. Do you think you could coordinate with @hashar to find a time we could actually dedicate to doing this?

• Jhernandez added a parent task: T155802: [EPIC] Build entire mobile.startup module using webpack bundler.Apr 11 2017, 3:13 PM

• Jhernandez mentioned this in T155802: [EPIC] Build entire mobile.startup module using webpack bundler.Apr 11 2017, 3:15 PM

We have a dependency on release engineering here. It would be great to coordinate to come up with a satisfactory solution instead of working on something just for ourselves.

Let's give some more time on the ticket for conversations.

Sure. My ask is that we talk to @greg and see if we can schedule some time in their busy schedule to work on this since this is causing us some problems.

Beside the quick chat early in March and the summary at T158980#3087905 , I do not plan to work on this nor do I have any idle cycles left to commit to it :-(

Jdlrobson added a parent task: T162936: Improve developer happiness in reading web maintained extensions.Apr 13 2017, 7:48 PM

phuedx removed a parent task: T160059: Popups technical improvements related to frontend assets.May 3 2017, 4:54 AM

The main tension here seems to be rebasing at time of merge.

I think the following are probably the best solutions to this problem:

Find some way to hook into the Gerrit rebase function and run npm run build at the end of it (not sure if this is possible)
Bring back everyone's favourite browser bot and get him to watch changes and rebase them himself.

In T158980#3295462, @Jdlrobson wrote:

Find some way to hook into the Gerrit rebase function and run npm run build at the end of it (not sure if this is possible)

This is not possible.

Jdlrobson renamed this task from Generate compiled assets from continuous integration to [EPIC] Generate compiled assets from continuous integration.Jun 2 2017, 8:31 PM

Jdlrobson added a project: Epic.

Per http://doodle.com/poll/88r2cm6tz649vvgk

We discussed https://etherpad.wikimedia.org/p/moderntoolingchain

greg moved this task from INBOX to Epics (ARCHIVED) on the Release-Engineering-Team board.Jun 13 2017, 11:51 PM

For the record, I sent an email to wikitech-l asking how does a build process look like for extensions to gather feedback: https://lists.wikimedia.org/pipermail/wikitech-l/2017-June/088264.html

Jdlrobson updated the task description. (Show Details)Jun 21 2017, 11:59 PM

Jdlrobson added a subscriber: thcipriani.

Nikerabbit subscribed.Jun 25 2017, 8:36 AM

• Jhernandez closed subtask T167181: Create bot that automatically rebases and rebuilds patches to master as Resolved.Jun 29 2017, 11:04 AM

Now we have a bot that helps merging how do people feel about compiling assets in other larger extensions..?

Seems like a topic for a different card.

Sure but what would that card look like? Are we ready to work on another extension? Which extension would it be?

Sent an email to discuss.

I replied to Reading-web-team] Using a bundler in another of our extensions
I suggest we revisit this in about a month.

• demon unsubscribed.Jul 7 2017, 7:03 AM

In T158980#3413033, @Jdlrobson wrote:

I replied to Reading-web-team] Using a bundler in another of our extensions
I suggest we revisit this in about a month.

The Reading-Web-Team mailing list is a private forum. This is a public forum. Could you summarise what you said here?

• Niedzielski subscribed.Aug 2 2017, 11:17 PM

In T158980#3414898, @phuedx wrote:

In T158980#3413033, @Jdlrobson wrote:

I replied to Reading-web-team] Using a bundler in another of our extensions
I suggest we revisit this in about a month.

The Reading-Web-Team mailing list is a private forum. This is a public forum. Could you summarise what you said here?

Ping :)

@greg sorry, I basically said: we're doing too many things at the task let's revisit this in a month (which is what I said in the comment)

To summarise the conversation that followed:

The build step is making the team's life a lot easier - easier to write, refactor and test code and opening up possibilities to innovate (in Popups we are now using ES6)
There were concerns MobileFrontend might not be the best extension to try this on next due to its size. RelatedArticles might be better.
The bot is a necessary evil. We don't want to rely on this, and are anxious about relying on it more, but we recognise to not rely on this, we need buy-in and support from release engineering to provide a build step and/or we need to skill up on understanding the integration config stack.

I think we're have more bandwidth to work on this later in the month/September.

• Jhernandez removed • Jhernandez as the assignee of this task.Aug 22 2017, 10:03 AM

Jdlrobson updated the task description. (Show Details)Aug 31 2017, 2:00 PM

Jdlrobson mentioned this in T173491: Change the extension of source-map files in webpack.config.js.

Jdlrobson mentioned this in T175082: mobile.startup should be only dependency of Minerva and Minerva should be able to define it itself.Sep 5 2017, 9:44 PM

Jdlrobson added a project: MobileFrontend (MobileFrontend and MinervaNeue architecture).Jan 12 2018, 5:16 PM

phuedx mentioned this in T160943: Automate WDQS GUI deployment.Mar 13 2018, 3:12 PM

FTR here's that email thread I sent to discuss last year: https://lists.wikimedia.org/pipermail/wikitech-l/2017-June/088264.html

• Niedzielski mentioned this in T190699: Repackage Minerva's JavaScript code with webpack.Mar 26 2018, 5:54 PM

We discussed this in passing a little today and mentioned that other repos such as the Wikipedia Portal and Wikidata have implemented solutions to this. Joaquin mentioned that these solutions, however, may not work well for MobileFrontend since it's expected to be treated as "just another extension" by CI.

• Mholloway subscribed.Apr 25 2018, 7:59 PM

Change 436327 had a related patch set uploaded (by Zfilipin; owner: Jdlrobson):
[mediawiki/extensions/Popups@master] Experiment: Build assets via MediaWiki hook

https://gerrit.wikimedia.org/r/436327

gerritbot added a project: Patch-For-Review.Jun 14 2018, 3:30 PM

Change 436327 abandoned by Jdlrobson:
Experiment: Build assets via MediaWiki hook

Reason:
Trying https://gerrit.wikimedia.org/r/441515

https://gerrit.wikimedia.org/r/436327

Jdlrobson mentioned this in T162936: Improve developer happiness in reading web maintained extensions.Jul 9 2018, 4:29 PM

Jdlrobson removed a project: Patch-For-Review.Jul 10 2018, 9:17 PM

Jdlrobson removed a project: Page-Previews.Jul 25 2018, 8:39 AM

Jdlrobson lowered the priority of this task from High to Low.Aug 30 2018, 1:30 AM

Jdlrobson moved this task from Backlog to Post-initial migration on the MobileFrontend (MobileFrontend and MinervaNeue architecture) board.

Jdlrobson removed a parent task: T155802: [EPIC] Build entire mobile.startup module using webpack bundler.Sep 6 2018, 4:32 PM

Jdlrobson raised the priority of this task from Low to Medium.Mar 8 2019, 8:04 PM

Jdlrobson moved this task from Post-initial migration to Misc on the MobileFrontend (MobileFrontend and MinervaNeue architecture) board.

hashar unsubscribed.Mar 25 2019, 3:59 PM

I believe this is blocked on T199004.

• Niedzielski added a subtask: T199004: RFC: Add a frontend build step to skins/extensions to our deploy process.Mar 29 2019, 6:51 PM

• Phabricator_maintenance edited projects, added Release-Engineering-Team-TODO; removed Release-Engineering-Team.Jun 12 2019, 11:40 PM

• Phabricator_maintenance moved this task from Should be empty (use Release-Engineering-Team) to Epics on the Release-Engineering-Team-TODO board.Jun 12 2019, 11:41 PM

greg added a project: Release-Engineering-Team.Jun 21 2019, 10:35 PM

Jdlrobson moved this task from Misc to Epics on the MobileFrontend (MobileFrontend and MinervaNeue architecture) board.Aug 2 2019, 5:03 PM

WMDE-leszek subscribed.Oct 18 2019, 12:04 PM

brennen mentioned this in T199004: RFC: Add a frontend build step to skins/extensions to our deploy process.Jan 7 2020, 7:41 PM

• Jhernandez unsubscribed.Apr 2 2020, 6:46 PM

• dcipoletti changed the status of subtask T199004: RFC: Add a frontend build step to skins/extensions to our deploy process from Open to Stalled.Jun 26 2020, 3:00 PM

• nnikkhoui mentioned this in T264244: Publish WVUI to NPM through CI.Oct 15 2020, 7:49 PM

Jdlrobson added a project: Deprecated-Design-Systems-team-board.Mar 22 2021, 9:28 PM

thcipriani edited projects, added Release-Engineering-Team (thcipriani-workboard-fiddling); removed Release-Engineering-Team, Release-Engineering-Team-TODO.Apr 20 2021, 3:42 AM

thcipriani moved this task from thcipriani-workboard-fiddling to Seen (ARCHIVE) on the Release-Engineering-Team board.Apr 20 2021, 3:58 AM

thcipriani edited projects, added Release-Engineering-Team; removed Release-Engineering-Team (thcipriani-workboard-fiddling).

thcipriani edited projects, added Release-Engineering-Team (Seen); removed Release-Engineering-Team.Apr 20 2021, 3:23 PM

We can't do immediate action to add a build step for Popups or other extension, that entirely depends on a solution that has to be found via the discussions being held as part of the RFC T199004. So dropping Release-Engineering-Team meanwhile.