Page MenuHomePhabricator
Create Task
Maniphest T173818

[Epic] Kill the Wikidata build step
Closed, ResolvedPublic
Actions

Assigned To
Addshore
Authored By
Addshore
Aug 22 2017, 11:11 AM
Tags
Referenced Files
None
Subscribers
Addshore
Agabi10
Aklapper
• Aleksey_WMDE
aude
bd808
daniel
View All 25 Subscribers
Tokens
"Party Time" token, awarded by Jdforrester-WMF."Meh!" token, awarded by bd808."Party Time" token, awarded by Smalyshev."Burninate" token, awarded by Liuxinyu970226."Love" token, awarded by Florian."Burninate" token, awarded by greg."Burninate" token, awarded by WMDE-leszek.

Description

This is a placeholder ticket for the creation of subtickets and planning of killing the Wikidata build step. (@Addshore will do this soon)

During Wikimania @Legoktm and @Addshore took a first stab at killing the build and essentially discovered all of the harder items & items that need discussion before we can move forward.

  • Review and classify all the composer dependencies as either libraries or extensions (https://etherpad.wikimedia.org/p/wikidatabuild)
  • T173936 Ensure all "libraries" don't interact with MediaWiki and are independent
  • T173942 Ensure all "extensions" are on Gerrit in mediawiki/extensions/...
  • T173941 Move libraries to mediawiki-vendor (Checking that they do not contain any mediawiki related codes) and remove them from the build.
  • T173940 Add extensions to make-wmf-branch script
  • T176948 Move loading of extensions from build into mediawiki-config
  • T174922 Decide what to do with the remaining JavaScript only stuff
  • T177087 Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
  • T177060 Sort out the extension lists and localization stuff for deployment
  • T175026 Deploy extensions as normal MW extensions (CommonSettings.php, etc.) and remove them from the build (note that we might need to set up manual classmaps for AutoloadClasses temporarily until T99865: Extension registration should support using a PSR-4 autoloader is resolved.
  • T181708 Undeploy the Wikidata extension / the build!!!!!!!!
  • T181706 Remove the labs instance & project used for creating the daily builds & puppet role
  • T181707 Remove the WikidataBuilder user from gerrit groups
  • T181709 Cleanup Phabricator tickets relating to the build

Details

SubjectRepoBranchLines +/-
Remove Wikidata build from make-wmf-branch scriptmediawiki/tools/releasemaster+0 -1
Remove Wikidata extensionintegration/configmaster+0 -2
Customize query in gerrit

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedAddshoreT173818 [Epic] Kill the Wikidata build step
ResolvedWMDE-leszekT173936 Ensure all wikidata "libraries" don't interact with MediaWiki and are independent
ResolvedAddshoreT173940 Add all wikidata build extensions to make-wmf-branch script
ResolvedAddshoreT173941 Add all wikidata build libraries to mediawiki-vendor repo
ResolvedAddshoreT173942 Move all wikidata build extensions to gerrit
ResolvedWMDE-leszekT174922 Decide what to do with Wikibase JS-only libraries regarding the build/deployment of Wikidata code
ResolvedAddshoreT175026 Remove extensions from the Wikidata build and deploy as a regular extension
DeclinedNoneT176408 Run MediaInfo tests on jenkins (post-merge job for Wikibase)
ResolvedAddshoreT176948 Move config & loading logic out of Wikidata build and into mediawiki-config
ResolvedAddshoreT177060 Stop using extension-list-wikidata from the build in mediawiki-config
ResolvedAddshoreT177087 Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
OpenNoneT179663 What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor?
ResolvedWMDE-leszekT180454 Sort out the mess that is the DataTypes extension / library / npm packages in order to kill the build
ResolvedLadsgroupT127292 [Task] Move DataTypes repository from Github to gerrit
ResolvedRelease• demonT178635 1.31.0-wmf.8 deployment blockers
ResolvedAddshoreT180727 WikibaseClient resources are broken in WMF
ResolvedAddshoreT181706 Kill the automated wikidata build on labs
ResolvedAddshoreT181707 Remove the WikidataBuilder user from gerrit groups
ResolvedAddshoreT181708 Undeploy the Wikidata extension
ResolvedAddshoreT181709 Cleanup Phabricator tickets once the build is dead
ResolvedWMDE-leszekT178083 Only run npm job on Jenkins for builds of data-values/value-view
ResolvedLadsgroupT181759 Figure out why WikidataBuild tests failed but Wikibase tests didnt in WMF CI
Resolvedbd808T181769 Update role::wikidata to work with new (no)build process
ResolvedAddshoreT181838 Mark extension-Wikidata & wikidata-build-resources on Gerrit as ARCHIVED
ResolvedAddshoreT182176 operations-puppet still uses the wikidata build in places
· · ·

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Addshore closed subtask T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer as Resolved.Nov 2 2017, 1:36 PM
WMDE-leszek updated the task description. (Show Details)Nov 2 2017, 1:36 PM
Addshore created subtask T179663: What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor?.Nov 3 2017, 10:53 AM
Addshore added a subscriber: Ladsgroup.Nov 3 2017, 11:00 AM

Update:

T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
Yesterday @WMDE-leszek & I merged all of the changes moving JavaScript around. This included a change to the wikidata build to make it load submodules from the Wikibase repo.
The JS libs that were loaded through composer are now loaded as submodules in the Wikibase extension.

T173941: Add all wikidata build libraries to mediawiki-vendor repo
Today and a bit of yesterday @Legoktm, @Ladsgroup and I started adding entries to mediawiki-vendor for all remaining wikibase composer libraries.
All of these should be merged by the end of the day.
The Wikidata development team now needs to keep the version in mediawiki-vendor up to date with what they need in production and we really need to make sure we don't break anything here.
I have filed T179663 to investigate what the current state is and see if we need to add any additional checks.

@greg as a result of the 2 paragraphs above the train next week will have all of this JS code moving around, as well as duplicate libraries for wikibase (only a short term thing).
Duplicate libraries should not cause any issues and is already the case with libraries such as assert which is in mediawiki-vendor and also the wikidata build.

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Addshore closed subtask T173941: Add all wikidata build libraries to mediawiki-vendor repo as Resolved.Nov 3 2017, 11:57 AM
greg added a comment.Nov 3 2017, 5:06 PM
In T173818#3732387, @Addshore wrote:

Update:

T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer
Yesterday @WMDE-leszek & I merged all of the changes moving JavaScript around. This included a change to the wikidata build to make it load submodules from the Wikibase repo.
The JS libs that were loaded through composer are now loaded as submodules in the Wikibase extension.

T173941: Add all wikidata build libraries to mediawiki-vendor repo
Today and a bit of yesterday @Legoktm, @Ladsgroup and I started adding entries to mediawiki-vendor for all remaining wikibase composer libraries.
All of these should be merged by the end of the day.
The Wikidata development team now needs to keep the version in mediawiki-vendor up to date with what they need in production and we really need to make sure we don't break anything here.
I have filed T179663 to investigate what the current state is and see if we need to add any additional checks.

@greg as a result of the 2 paragraphs above the train next week will have all of this JS code moving around, as well as duplicate libraries for wikibase (only a short term thing).
Duplicate libraries should not cause any issues and is already the case with libraries such as assert which is in mediawiki-vendor and also the wikidata build.

ping @demon :)

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Yeah, let's do that, when makes sense? also cc @demon :)

• demon added a comment.Nov 3 2017, 5:12 PM
In T173818#3733644, @greg wrote:
In T173818#3732387, @Addshore wrote:

@greg as a result of the 2 paragraphs above the train next week will have all of this JS code moving around, as well as duplicate libraries for wikibase (only a short term thing).
Duplicate libraries should not cause any issues and is already the case with libraries such as assert which is in mediawiki-vendor and also the wikidata build.

ping @demon :)

Ack'd. Yeah, this shouldn't be problematic but good to point out / keep an eye on.

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Yeah, let's do that, when makes sense? also cc @demon :)

Yeah, that makes sense to not steal all the SWAT windows. We could maybe piggyback on the train windows since I almost never need the full 2 hours?

Addshore added a comment.Nov 3 2017, 5:17 PM
In T173818#3733667, @demon wrote:

I plan on tackling T176948 and T177060 next week ahead of taking extensions out of the build along with any other config cleanups that we may need.
Then we will start the process of loading extensions from outside the build.
All extensions have already been added to the extension lists for some time now, I'll come up with some chain of config patches to get us to the end at the start of next week.
@greg I am guessing we will want to schedule our own 'special' deploy slots for this rather that using SWAT?

Yeah, let's do that, when makes sense? also cc @demon :)

Yeah, that makes sense to not steal all the SWAT windows. We could maybe piggyback on the train windows since I almost never need the full 2 hours?

Piggybacking on the train windows could work well.
I can also do things in EU day time to avoid clashing with other slots / busy periods.

Addshore closed subtask T176948: Move config & loading logic out of Wikidata build and into mediawiki-config as Resolved.Nov 8 2017, 6:36 PM
Addshore updated the task description. (Show Details)
Addshore closed subtask T179663: What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor? as Resolved.Nov 9 2017, 8:25 AM
Liuxinyu970226 awarded a token.Nov 9 2017, 10:07 AM
Addshore added a subtask: T177060: Stop using extension-list-wikidata from the build in mediawiki-config.Nov 9 2017, 2:08 PM
Addshore updated the task description. (Show Details)Nov 9 2017, 3:52 PM
WMDE-leszek reopened subtask T173941: Add all wikidata build libraries to mediawiki-vendor repo as Open.Nov 14 2017, 11:21 AM
WMDE-leszek reopened subtask T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer as Open.
Addshore created subtask T180454: Sort out the mess that is the DataTypes extension / library / npm packages in order to kill the build.Nov 14 2017, 12:34 PM
Addshore closed subtask T173941: Add all wikidata build libraries to mediawiki-vendor repo as Resolved.Nov 14 2017, 2:54 PM
Addshore closed subtask T177087: Make Wikibase-related JS libraries npm packages and include them as git submodules in Wikibase.git instead of installing as PHP libs using Composer as Resolved.Nov 14 2017, 4:02 PM
• demon mentioned this in T180539: wmf.8 still on old Wikidata branch.Nov 15 2017, 12:10 AM
Addshore reopened subtask T179663: What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor? as Open.Nov 15 2017, 8:47 PM
Addshore added a subtask: T178635: 1.31.0-wmf.8 deployment blockers.Nov 16 2017, 10:26 AM
Addshore added a subtask: T180727: WikibaseClient resources are broken in WMF.Nov 16 2017, 9:11 PM
Smalyshev subscribed.Nov 17 2017, 3:11 AM
Addshore closed subtask T180727: WikibaseClient resources are broken in WMF as Resolved.Nov 17 2017, 5:44 PM
Izno subscribed.Nov 21 2017, 11:04 PM
• demon closed subtask T178635: 1.31.0-wmf.8 deployment blockers as Resolved.Nov 22 2017, 11:31 PM
gerritbot subscribed.Nov 30 2017, 12:02 PM

Change 394288 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/tools/release@master] DNM Remove Wikidata build from make-wmf-branch script

https://gerrit.wikimedia.org/r/394288

gerritbot added a project: Patch-For-Review.Nov 30 2017, 12:02 PM
Addshore created subtask T181706: Kill the automated wikidata build on labs.Nov 30 2017, 12:27 PM
Addshore updated the task description. (Show Details)
Addshore created subtask T181707: Remove the WikidataBuilder user from gerrit groups.Nov 30 2017, 12:30 PM
Addshore updated the task description. (Show Details)
Addshore created subtask T181708: Undeploy the Wikidata extension.
Addshore updated the task description. (Show Details)
Addshore updated the task description. (Show Details)Nov 30 2017, 12:33 PM
Addshore created subtask T181709: Cleanup Phabricator tickets once the build is dead.
Addshore updated the task description. (Show Details)Nov 30 2017, 12:37 PM
Stashbot subscribed.Nov 30 2017, 8:30 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:30:40Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (duration: 00m 47s)

Stashbot added a comment.Nov 30 2017, 8:32 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:32:30Z] <addshore@tin> Synchronized wmf-config: REVERT wdbuild: T173818: add switch to ease killing (duration: 00m 47s)

Stashbot added a comment.Nov 30 2017, 8:59 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T20:59:44Z] <addshore@tin> Synchronized wmf-config/InitialiseSettings.php: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 45s)

Stashbot added a comment.Nov 30 2017, 9:01 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T21:01:28Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818: add switch to ease killing (again) (duration: 00m 46s)

Addshore created subtask T181759: Figure out why WikidataBuild tests failed but Wikibase tests didnt in WMF CI.Nov 30 2017, 9:45 PM
Stashbot added a comment.Nov 30 2017, 11:21 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:21:54Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 [[gerrit:394214|wdbuild: Stop loading from build on ALL WIKIS]] The build is dead! Mwahahaaa (duration: 00m 47s)

Addshore closed subtask T175026: Remove extensions from the Wikidata build and deploy as a regular extension as Resolved.Nov 30 2017, 11:26 PM
Addshore updated the task description. (Show Details)
Stashbot added a comment.Nov 30 2017, 11:29 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:29:02Z] <addshore@tin> Synchronized wmf-config/Wikibase-buildentry.php: wdbuild: T173818 [[gerrit:394215|Remove wmgUseWikidataBuild]] (duration: 00m 45s)

Stashbot added a comment.Nov 30 2017, 11:30 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:30:38Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 [[gerrit:394215|Remove wmgUseWikidataBuild]] (duration: 00m 46s)

Stashbot added a comment.Nov 30 2017, 11:34 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:34:33Z] <addshore@tin> Synchronized wmf-config/Wikibase.php: wdbuild: T173818 [[gerrit:394216|Remove Wikibase-buildentry.php config file (empty)]] (duration: 00m 45s)

Stashbot added a comment.Nov 30 2017, 11:35 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:35:35Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 [[gerrit:394216|Remove Wikibase-buildentry.php config file (empty)]] (duration: 00m 46s)

Addshore closed subtask T177060: Stop using extension-list-wikidata from the build in mediawiki-config as Resolved.Nov 30 2017, 11:38 PM
Stashbot added a comment.Nov 30 2017, 11:39 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-30T23:39:12Z] <addshore@tin> Synchronized wmf-config: wdbuild: T173818 T177060 [[gerrit:394282|Add wikidata extensions to extension-list]] (duration: 00m 46s)

Stashbot mentioned this in T177060: Stop using extension-list-wikidata from the build in mediawiki-config.Nov 30 2017, 11:39 PM
Addshore added a comment.Nov 30 2017, 11:44 PM

The build is now DEAD! 🗡
All that remains is cleanup 🍰

Smalyshev awarded a token.Nov 30 2017, 11:47 PM
bd808 awarded a token.Nov 30 2017, 11:56 PM
bd808 subscribed.Nov 30 2017, 11:58 PM

The MediaWiki-Vagrant role clones https://gerrit.wikimedia.org/r/wikidata/build-resources. Does that need some replacement as well?

Jdforrester-WMF updated the task description. (Show Details)Dec 1 2017, 12:03 AM
bd808 mentioned this in T181769: Update role::wikidata to work with new (no)build process.Dec 1 2017, 12:11 AM
bd808 added a subtask: T181769: Update role::wikidata to work with new (no)build process.
Ladsgroup closed subtask T181759: Figure out why WikidataBuild tests failed but Wikibase tests didnt in WMF CI as Resolved.Dec 1 2017, 11:36 AM
Addshore created subtask T181838: Mark extension-Wikidata & wikidata-build-resources on Gerrit as ARCHIVED.Dec 1 2017, 7:05 PM
Addshore closed subtask T181707: Remove the WikidataBuilder user from gerrit groups as Resolved.Dec 1 2017, 7:08 PM
Addshore added a subtask: T178083: Only run npm job on Jenkins for builds of data-values/value-view.Dec 1 2017, 7:23 PM
Addshore closed subtask T181709: Cleanup Phabricator tickets once the build is dead as Resolved.
Addshore updated the task description. (Show Details)
Jdforrester-WMF awarded a token.Dec 1 2017, 8:41 PM
gerritbot added a comment.Dec 5 2017, 5:48 PM

Change 395581 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[integration/config@master] Remove Wikidata extension

https://gerrit.wikimedia.org/r/395581

Addshore added a subtask: T182159: dispatchChanges.php still being run from the Wikidata build.Dec 6 2017, 8:28 AM
Addshore removed a subtask: T182159: dispatchChanges.php still being run from the Wikidata build.Dec 6 2017, 8:40 AM
gerritbot added a comment.Dec 6 2017, 9:44 AM

Change 395581 merged by jenkins-bot:
[integration/config@master] Remove Wikidata extension

https://gerrit.wikimedia.org/r/395581

Addshore closed subtask T181706: Kill the automated wikidata build on labs as Resolved.Dec 6 2017, 10:15 AM
Addshore closed subtask T182176: operations-puppet still uses the wikidata build in places as Resolved.Dec 6 2017, 3:12 PM
Addshore closed subtask T180454: Sort out the mess that is the DataTypes extension / library / npm packages in order to kill the build as Resolved.Dec 8 2017, 10:25 AM
bd808 closed subtask T181769: Update role::wikidata to work with new (no)build process as Resolved.Jan 1 2018, 4:36 AM
Addshore closed this task as Resolved.Jan 2 2018, 4:54 PM
Addshore updated the task description. (Show Details)

Wheeee =]

gerritbot added a comment.Jan 2 2018, 6:15 PM

Change 394288 merged by jenkins-bot:
[mediawiki/tools/release@master] Remove Wikidata build from make-wmf-branch script

https://gerrit.wikimedia.org/r/394288

Addshore moved this task from Back Burner 🏛️ to Closing ✔️ on the User-Addshore board.Jan 2 2018, 6:55 PM
Liuxinyu970226 unsubscribed.Jan 5 2018, 11:30 AM
Liuxinyu970226 subscribed.
Liuxinyu970226 unsubscribed.
Addshore closed subtask T181708: Undeploy the Wikidata extension as Resolved.Jan 9 2018, 7:43 AM
Addshore closed subtask T181838: Mark extension-Wikidata & wikidata-build-resources on Gerrit as ARCHIVED as Resolved.Feb 7 2018, 7:54 PM
WMDE-leszek closed subtask T178083: Only run npm job on Jenkins for builds of data-values/value-view as Resolved.Feb 26 2018, 4:09 PM
Krinkle mentioned this in T189538: Reduce complexity of Wikibase JS module registration.Mar 12 2018, 10:53 PM
Addshore closed subtask T176408: Run MediaInfo tests on jenkins (post-merge job for Wikibase) as Declined.Aug 20 2018, 3:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL