Page MenuHomePhabricator

[Task] Split wikibase.git
Closed, DeclinedPublic

Assigned To
Authored By
Nov 25 2014, 12:14 PM
Referenced Files
F22373: DSC_0468.JPG
Dec 18 2014, 11:12 AM
F17944: IMG_20141202_143956_2.jpg
Dec 2 2014, 2:09 PM
"Like" token, awarded by Liuxinyu970226."Like" token, awarded by JeroenDeDauw."Like" token, awarded by Addshore.


This is tracking steps for splitting wikibase.git.

Jeroen De Dauw, 13 July 2014, "Splitting the Wikibase git repo":

Now that we have code that in some way depends on things in the Wikibase git (ie Wikibase Query depends on Wikibase Repo and Wikibase Lib), the state of this git repo is becoming problematic. It forces development against master and pulling in of both repo and client.

To address the most immediate problems, I suggest the following steps:

  • We create a 0.5 tag, which Wikibase Query and co can then use
  • We split the git repo in 3: repo, client and lib
  • The new components get bumped to 0.6 alpha
  • Wikibase Query and co reference the new components when we get to it

This is not blocked by, or blocks, further cleanup of repo and client, and the further breakup and phaseout of lib.

Things to consider before proceeding:

  • We do not want to do this close before the branch point, preferably close after one
  • This will change how things are installed (no more need for these silly wgEnableWikibaseRepo and wgEnableWikibaseClient settings). Not sure this requires work from WMF side, since we got the build. Build and install docs will need updating though
  • We will need to migrate commits awaiting review. Commits that span across the different components will require the most work. So getting the review queue down and avoiding big commits is helpful

Jeroen De Dauw, 28 October 2014, "Testing the Wikibase.git split":

Last week I created an experimental split of the Wikibase.git repo, as previously discussed. Before doing this for real, I'd like some help with testing, and ensuring we are all good to go.

These repos are at:

(They are on GH since it's easier to create and delete repos there - the real split will be on Gerrit.)

The PHPUnit tests run fine for me. No idea about the Selenium or QUnit tests (this is mainly what I want help with).

There is one known blocker at present, which is an issue with the paths of the resource loader modules in various of our components. These components determine their location relative to the extensions directory [0], which obviously breaks if they are not in the extensions directory. When installing them in the wiki root, they end up at wikiroot/vendor/[ValueViewPath], end thus break. Currently this is avoided because they are installed in Wikibase.git, end thus end up in wikiroot/extensions/Wikibase/vendor/[ValueViewPath]. We need to update these components to either figure out the path correctly in both cases or mark them as MediaWiki extensions rather than libraries, so they end up in wikiroot/extensions/[ValueView]. Adrian told me he is already running into this problem and has some local hacks. I'd appreciate input from the people working on these JS components on how to proceed.

The path issue will not occur for deployment, since the paths will end up in wikiroot/extensions due to our build step. It is however a blocker since else people would be force to use the build step during development, which is silly. For testing purposes I've created this "extension" that you can include in LocalSettings.php as shown in the README file, and then run "composer install" in it to pull in Repo and Client.


Related Objects


Event Timeline

Lydia_Pintscher assigned this task to Wikidata-bugs.
Lydia_Pintscher raised the priority of this task from to High.
Lydia_Pintscher updated the task description. (Show Details)
Lydia_Pintscher changed Security from none to None.
Lydia_Pintscher subscribed.
JanZerebecki lowered the priority of this task from High to Medium.Nov 25 2014, 12:44 PM
Lydia_Pintscher removed a subscriber: Unknown Object (MLST).

Result of today's discussion is that the next step in the split will be:

IMG_20141202_143956_2.jpg (2×2 px, 392 KB)

Lydia_Pintscher moved this task from incoming to hold on the Wikidata board.

Change 225080 had a related patch set uploaded (by Paladox):
Add .gitreview

Change 225086 had a related patch set uploaded (by Paladox):
Add tests for WikibaseRepository

Change 225087 had a related patch set uploaded (by Paladox):
Add support for running test this extension needs once it is being used

Change 225086 had a related patch set uploaded (by Paladox):
Add tests for WikibaseRepository

@Paladox Thanks for looking into this. Please mind the numerous blockers though. Please consult with the team before actually moving code, that can get quite disruptive. Thanks again!

Ok I am only setting the repo up for now ready for when you start to move the data over.

Jenkings is failing beause it carnt merge. Master branch carnt be found.

To copy It over you can follow

Could you copy over the data to repo so that we can begin to work on it mergin changes from the wikibase branch if they change the repo branch and then do it for every repo for example after repo is done do it to lib then client and any other repo that needs doing.

That is quite an involved job, which is why we are not done with it yet. It requires making the current code more modular, so that in can be split in two separate repos, so it is not just copying the repo. If I remember correctly we decided we would get rid of lib instead of splitting lib off. Thus that needs to be done first. See this tasks subtasks. It requires quite a bit of understanding of the Wikibase code. It may not a good task to do while starting on the Wikibase repo.

Change 225086 abandoned by Hashar:
Add tests for WikibaseRepository

No point in keeping this change around. There is a lot more work to be accomplished before.

Change 225087 abandoned by Paladox:
Add support for running test this extension needs once it is being used

Addshore renamed this task from split wikibase.git to [Task] Split wikibase.git.Sep 2 2015, 5:06 PM

Another reason for splitting >
Wikibase.git is getting rather big....

I wonder if we could move forward on this task that has been open for a long while any time soon!

When we first discussed this I brought up that blocking this task on removing Wikibase Lib would essentially defer it to the end of time. At that point I was under the impression we where very slowly decreasing the size of Lib. Turns out that was too optimistic:

Now there are more extensions that build on top of Client or Repository, we have even more incentive to use distinct git repositories.

If I could give this another thumbs up I would...

@thiemowmde can you please look at the subtasks of this and see which ones can be closed?

I briefly looked into splitting the Git repository a while ago (I wasn’t aware of this task at the time); copy+pasting the relevant bits here:

For reference, over the past two years, 75% of Wikibase commits only touched one subdirectory, 14% touched two, 6% touched three, and 4% touched four or more (or files in the root directory). That’s excluding merge commits and l10n-bot stuff; see P8544 for details.

Some of these will be submodule updates (we have six submodules in view/lib/), but still, this seems to suggest that the Wikibase extensions aren’t as tightly coupled as I thought, and that splitting them into separate repositories might actually be an option.

Re-running the script today, the percentage of commits touching only one subdirectory is a bit higher (78%) – I suspect this is thanks to the work on data bridge and tainted references.

So, the old thinking here was that it would make sense to split this into client lib and repo.
That however is from a time before compose (for us / wmf).

The more recent discussions (over the past 1-2 years) have been regarding moving lib into either client or repo, and splitting, making one depend on the other.
I don't remember while way @daniel thought this would make the most sense to happen.
I also don't appear to be able to find anything on phab documenting this.

My thoughts here are:

  • You can have a repo without a client (none of the client features)
  • You can't have a client without a repo and without access to data from a repo somewhere
  • So perhaps client should depend on repo.

But I seem to remember @daniel was leaning the other way

I'll try ot poke him and also dig up any previous notes.

I might be missing some history here but as far as I can understand this ticket has been rendered obsolete thanks to Wikibase Extension Decoupling and Registration.
@Tarrow @Lucas_Werkmeister_WMDE @Ladsgroup @ItamarWMDE ?

In one discussion in the team we agreed to decline this in favor of monorepo approach once the ADR is accepted (still in the process). I think we can leave it until the ADR is accepted, not much activity happening on it anyway :D

The ADR hasn’t made any progress but I still think we can close this.