Page MenuHomePhabricator

Move portal deployment artifacts into their own repo
Closed, ResolvedPublic

Description

Currently the wikimedia-portals repo contains all the source files and build artifacts for the wikimedia portals.
The build artifacts change frequently because of a cache-busting hash on the build file names, and running a build effectively deletes and adds hundreds of files.
https://phabricator.wikimedia.org/rWPOR3862e2f5c2c3196872b960a6e63cc5f1a70e9a2b
Because of this, most Gerrit patches are only submitted with the source files to make it clear what the functional changes are. Then, occasionally, a patch with a build is submitted that changes the build artifacts.

Placing the build artifacts in their own repo would make it easier to track the portal builds and lighten the source repo significantly. The repo with the build artifacts can then be included into the source repo as a submodule.

Steps

  • create a new wikimedia/portals/deploy repository in Gerrit
  • replace the prod folder in the wikimedia-portals repo with a submodule of wikimedia-portals-deploy (393223, 393224)
  • move the files sync-portals and urls-to-purge.txt into the root of the wikimedia-portals-deployrepo
  • change line 5 in sync-portals
scap sync-file portals/prod/wikipedia.org/assets $*
to 
scap sync-file portals/wikipedia.org/assets $*
  • change the portals submodule in mediawiki-config from wikimedia-portals to wikimedia-portals-deploy(393239)
  • change the following symlink in mediawiki-config (393239)
 /mediawiki-config/docroot/wwwportal/portal@ -> ../../portals/prod 
to 
/mediawiki-config/docroot/wwwportal/portal@ -> ../../portals
  • Set the Jenkins job to commit the build assets from portals to the wikimedia/portals/deploy repo (393252)

Event Timeline

Restricted Application added a project: Discovery. ยท View Herald TranscriptNov 17 2017, 9:50 AM
Restricted Application added a subscriber: Aklapper. ยท View Herald Transcript

Mentioned in SAL (#wikimedia-releng) [2017-11-21T13:08:31Z] <hashar> gerrit: created wikimedia/portals/deploy https://gerrit.wikimedia.org/r/#/admin/projects/wikimedia/portals/deploy for jan_drewniak | T180777

hashar updated the task description. (Show Details)Nov 21 2017, 1:08 PM

Just to be clear about the submodule setup, as I was discussing with @hashar today, I think it should look like this:

portals-repo/
- data
- build-scripts, etc...
- dev
  |- wikipedia.org <-(source templates)
- prod <-(portals/deploy submodule)
  |- wikipedia.org<-(compiled assets)
  |- wikibooks.org
  |- etc.. 
  |- sync-portals
  |- urls-to-purge.txt

Where just the compiled assets and a few scripts (sync-portals, urls-to-purge.txt) would live in the deploy repository.
Then in mediawiki-config, we replace the portals repository with the portals/deploy repo:

mediawiki-config/ 
- portals <-(portals/deploy submodule)
  |- wikipedia.org <-(compiled assets)
  |- wikibooks.org
  |- etc.. 
  |- sync-portals
  |- urls-to-purge.txt

How this will effect the build process
With this submodule setup, the process for submitting a new build to Gerrit will change slightly.
Instead of submitting the build as a patch to the source repo, it will be submitted as a patch to the deploy repo.
So instead of:
checkout portals -> run build -> commit to portals
it would be
checkout portals -> run build -> cd into submodule -> commit to portals/deploy
For this to work in Jenkins, it would only require a minor modification (cd prod and commit from there).
For this to work on developer machines, users will have to change the git remote origin inside the submodule from https to ssh to push to Gerrit (I'll include a helper npm script that can do this).

Change 393223 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] [WIP] Adding portals/deploy repo as submodule

https://gerrit.wikimedia.org/r/393223

Change 393224 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals@master] [WIP] Initializing portals/deploy submodule

https://gerrit.wikimedia.org/r/393224

Change 393239 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[operations/mediawiki-config@master] [WIP] Replace portals submodule with portals/deploy submodule

https://gerrit.wikimedia.org/r/393239

Change 393252 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[integration/config@master] [WIP] Setting portals JJ to commit to portals/deploy

https://gerrit.wikimedia.org/r/393252

Jdrewniak updated the task description. (Show Details)Nov 24 2017, 4:02 PM
Jdrewniak updated the task description. (Show Details)Nov 24 2017, 4:08 PM

Change 394539 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals/deploy@master] Adding selenium tests

https://gerrit.wikimedia.org/r/394539

Change 394565 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: add Chromedriver to npm-browser-test

https://gerrit.wikimedia.org/r/394565

Change 394565 merged by jenkins-bot:
[integration/config@master] docker: add Chromedriver to npm-browser-test

https://gerrit.wikimedia.org/r/394565

Change 414970 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Move job from experimental to test and gate-and-submit pipelines for wikimedia/portals/deploy

https://gerrit.wikimedia.org/r/414970

Change 414970 merged by jenkins-bot:
[integration/config@master] Move job from experimental to test and gate-and-submit pipelines for wikimedia/portals/deploy

https://gerrit.wikimedia.org/r/414970

@Jdrewniak looks like this task is complete?

Change 394539 merged by jenkins-bot:
[wikimedia/portals/deploy@master] Adding selenium tests

https://gerrit.wikimedia.org/r/394539

Change 415291 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[wikimedia/portals/deploy@master] Adding scap script and urls-to-purge.txt to repo

https://gerrit.wikimedia.org/r/415291

Jdrewniak updated the task description. (Show Details)Feb 28 2018, 2:02 PM

Change 415291 merged by jenkins-bot:
[wikimedia/portals/deploy@master] Adding scap script and urls-to-purge.txt to repo

https://gerrit.wikimedia.org/r/415291

Change 393223 merged by jenkins-bot:
[wikimedia/portals@master] Adding portals/deploy repo as submodule

https://gerrit.wikimedia.org/r/393223

Jdrewniak updated the task description. (Show Details)Feb 28 2018, 2:34 PM

Change 393224 merged by jenkins-bot:
[wikimedia/portals@master] Initializing portals/deploy submodule

https://gerrit.wikimedia.org/r/393224

ok! I'm on a roll here. The selenium tests are working. The portal/deploy repo has been submodule'd into the src repo, there's just a few outstanding config/jenkins things left.
@hashar I've updated this patch that run the portalsBuilder in the deploy repository https://gerrit.wikimedia.org/r/#/c/393252/.
After that, this config patch will have to be updated to point to the new portal/deploy submodule https://gerrit.wikimedia.org/r/#/c/393239/

debt added a comment.Mar 9 2018, 5:27 PM

Woohoo - glad to see this progressing (even if I'm late to the party to see it!) :)

pinging @hashar, do you have time to review https://gerrit.wikimedia.org/r/#/c/393252/ ?
That patch sets the portalBuilder job to push a commit to the portals/deploy repository, instead of portals.

Eek sorry I have missed it. I have reconfigured the job and triggered a build:

hashar assigned this task to Jdrewniak.Mar 13 2018, 12:17 PM

Jan did all the work. I have just been pushing buttons.

Change 393252 merged by jenkins-bot:
[integration/config@master] Setting portals JJ to commit to portals/deploy

https://gerrit.wikimedia.org/r/393252

Jdrewniak updated the task description. (Show Details)Mar 13 2018, 3:50 PM

Change 393239 merged by jenkins-bot:
[operations/mediawiki-config@master] Replace portals submodule with portals/deploy submodule

https://gerrit.wikimedia.org/r/393239

Mentioned in SAL (#wikimedia-operations) [2018-03-19T11:13:26Z] <jdrewniak@tin> Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:393239|Switching portals submodule to portals-deploy (T180777)]] (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2018-03-19T11:14:25Z] <jdrewniak@tin> Synchronized portals: Wikimedia Portals Update: [[gerrit:393239|Switching portals submodule to portals-deploy (T180777)]] (duration: 00m 58s)

Jdrewniak updated the task description. (Show Details)Mar 19 2018, 11:43 AM
Jdrewniak closed this task as Resolved.Apr 23 2018, 10:13 AM