Page MenuHomePhabricator

[Epic] Kill the Wikidata build step
Closed, ResolvedPublic

Tokens
"Party Time" token, awarded by Jdforrester-WMF."Meh!" token, awarded by bd808."Party Time" token, awarded by Smalyshev."Burninate" token, awarded by Liuxinyu970226."Love" token, awarded by Florian."Burninate" token, awarded by greg."Burninate" token, awarded by WMDE-leszek.
Assigned To
Authored By
Addshore, Aug 22 2017

Description

This is a placeholder ticket for the creation of subtickets and planning of killing the Wikidata build step. (@Addshore will do this soon)

During Wikimania @Legoktm and @Addshore took a first stab at killing the build and essentially discovered all of the harder items & items that need discussion before we can move forward.

  • Review and classify all the composer dependencies as either libraries or extensions (https://etherpad.wikimedia.org/p/wikidatabuild)
  • T173936 Ensure all "libraries" don't interact with MediaWiki and are independent
  • T173942 Ensure all "extensions" are on Gerrit in mediawiki/extensions/...
  • T173941 Move libraries to mediawiki-vendor (Checking that they do not contain any mediawiki related codes) and remove them from the build.
  • T173940 Add extensions to make-wmf-branch script
  • T176948 Move loading of extensions from build into mediawiki-config
  • T174922 Decide what to do with the remaining JavaScript only stuff
  • T177087 Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
  • T177060 Sort out the extension lists and localization stuff for deployment
  • T175026 Deploy extensions as normal MW extensions (CommonSettings.php, etc.) and remove them from the build (note that we might need to set up manual classmaps for AutoloadClasses temporarily until T99865: Extension registration should support using a PSR-4 autoloader is resolved.
  • T181708 Undeploy the Wikidata extension / the build!!!!!!!!
  • T181706 Remove the labs instance & project used for creating the daily builds & puppet role
  • T181707 Remove the WikidataBuilder user from gerrit groups
  • T181709 Cleanup Phabricator tickets relating to the build

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusAssignedTask
Resolved Addshore
ResolvedWMDE-leszek
Resolved Addshore
Resolved Addshore
Resolved Addshore
ResolvedWMDE-leszek
Resolved Addshore
DeclinedNone
Resolved Addshore
Resolved Addshore
Resolved Addshore
OpenNone
ResolvedWMDE-leszek
ResolvedLadsgroup
Resolveddemon
Resolved Addshore
Resolved Addshore
Resolved Addshore
Resolved Addshore
Resolved Addshore
ResolvedWMDE-leszek
ResolvedLadsgroup
Resolvedbd808
Resolved Addshore
Resolved Addshore

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
WMDE-leszek updated the task description. (Show Details)Nov 2 2017, 1:36 PM

Update:

T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
Yesterday @WMDE-leszek & I merged all of the changes moving JavaScript around. This included a change to the wikidata build to make it load submodules from the Wikibase repo.
The JS libs that were loaded through composer are now loaded as submodules in the Wikibase extension.

T173941: Add all wikidata build libraries to mediawiki-vendor repo
Today and a bit of yesterday @Legoktm, @Ladsgroup and I started adding entries to mediawiki-vendor for all remaining wikibase composer libraries.
All of these should be merged by the end of the day.
The Wikidata development team now needs to keep the version in mediawiki-vendor up to date with what they need in production and we really need to make sure we don't break anything here.
I have filed T179663 to investigate what the current state is and see if we need to add any additional checks.

@greg as a result of the 2 paragraphs above the train next week will have all of this JS code moving around, as well as duplicate libraries for wikibase (only a short term thing).
Duplicate libraries should not cause any issues and is already the case with libraries such as assert which is in mediawiki-vendor and also the wikidata build.

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

greg added a comment.Nov 3 2017, 5:06 PM

Update:
T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
Yesterday @WMDE-leszek & I merged all of the changes moving JavaScript around. This included a change to the wikidata build to make it load submodules from the Wikibase repo.
The JS libs that were loaded through composer are now loaded as submodules in the Wikibase extension.
T173941: Add all wikidata build libraries to mediawiki-vendor repo
Today and a bit of yesterday @Legoktm, @Ladsgroup and I started adding entries to mediawiki-vendor for all remaining wikibase composer libraries.
All of these should be merged by the end of the day.
The Wikidata development team now needs to keep the version in mediawiki-vendor up to date with what they need in production and we really need to make sure we don't break anything here.
I have filed T179663 to investigate what the current state is and see if we need to add any additional checks.
@greg as a result of the 2 paragraphs above the train next week will have all of this JS code moving around, as well as duplicate libraries for wikibase (only a short term thing).
Duplicate libraries should not cause any issues and is already the case with libraries such as assert which is in mediawiki-vendor and also the wikidata build.

ping @demon :)

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Yeah, let's do that, when makes sense? also cc @demon :)

demon added a comment.Nov 3 2017, 5:12 PM

@greg as a result of the 2 paragraphs above the train next week will have all of this JS code moving around, as well as duplicate libraries for wikibase (only a short term thing).
Duplicate libraries should not cause any issues and is already the case with libraries such as assert which is in mediawiki-vendor and also the wikidata build.

ping @demon :)

Ack'd. Yeah, this shouldn't be problematic but good to point out / keep an eye on.

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Yeah, let's do that, when makes sense? also cc @demon :)

Yeah, that makes sense to not steal all the SWAT windows. We could maybe piggyback on the train windows since I almost never need the full 2 hours?

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Yeah, let's do that, when makes sense? also cc @demon :)

Yeah, that makes sense to not steal all the SWAT windows. We could maybe piggyback on the train windows since I almost never need the full 2 hours?

Piggybacking on the train windows could work well.
I can also do things in EU day time to avoid clashing with other slots / busy periods.

Addshore updated the task description. (Show Details)Nov 9 2017, 3:52 PM
Izno added a subscriber: Izno.Nov 21 2017, 11:04 PM

Change 394288 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/tools/release@master] DNM Remove Wikidata build from make-wmf-branch script

https://gerrit.wikimedia.org/r/394288

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:30:40Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:32:30Z] <addshore@tin> Synchronized wmf-config: REVERT wdbuild: T173818: add switch to ease killing (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:59:44Z] <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T21:01:28Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:21:54Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 [[gerrit:394214|wdbuild: Stop loading from build on ALL WIKIS]] The build is dead! Mwahahaaa (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:29:02Z] <addshore@tin> Synchronized wmf-config/Wikibase-buildentry.php: wdbuild: T173818 [[gerrit:394215|Remove wmgUseWikidataBuild]] (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:30:38Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 [[gerrit:394215|Remove wmgUseWikidataBuild]] (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:34:33Z] <addshore@tin> Synchronized wmf-config/Wikibase.php: wdbuild: T173818 [[gerrit:394216|Remove Wikibase-buildentry.php config file (empty)]] (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:35:35Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 [[gerrit:394216|Remove Wikibase-buildentry.php config file (empty)]] (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:39:12Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 T177060 [[gerrit:394282|Add wikidata extensions to extension-list]] (duration: 00m 46s)

The build is now DEAD! 🗡
All that remains is cleanup 🍰

bd808 added a subscriber: bd808.Nov 30 2017, 11:58 PM

The MediaWiki-Vagrant role clones https://gerrit.wikimedia.org/r/wikidata/build-resources. Does that need some replacement as well?

Change 395581 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[integration/config@master] Remove Wikidata extension

https://gerrit.wikimedia.org/r/395581

Change 395581 merged by jenkins-bot:
[integration/config@master] Remove Wikidata extension

https://gerrit.wikimedia.org/r/395581

Addshore closed this task as Resolved.Jan 2 2018, 4:54 PM
Addshore updated the task description. (Show Details)

Wheeee =]

Change 394288 merged by jenkins-bot:
[mediawiki/tools/release@master] Remove Wikidata build from make-wmf-branch script

https://gerrit.wikimedia.org/r/394288

Liuxinyu970226 added a subscriber: Liuxinyu970226.
Liuxinyu970226 removed a subscriber: Liuxinyu970226.