Page MenuHomePhabricator

Support nested package.json files
Open, Needs TriagePublic

Description

As part of our Wikidata-Bridge work, we’ve set up a node project in a subdirectory of the Wikibase Git repository, with a package.json at client/data-bridge/package.json. As far as I can tell, LibraryUpgrader currently won’t update this project. It would be nice if it could do that.

We’re not the only ones to do this either; with code search, I can also see some nested package.json files in other projects. However, as far as I can tell only one other project would benefit from this (VisualEditor’s rebaser, cc @Esanders), so this probably shouldn’t be autodiscovered (i. e. upgrade every **/package.json) – a manually maintained list of extra package.json files would probably be enough.

Event Timeline

Is the node project subdirectory meant to be a short term thing or expected to stay around for quite a while?

@Krinkle , @Jdforrester-WMF and I had discussed this previously (2019-06-10 in #wikimedia-releng):

[16:19:24] <legoktm>	 James_F, Krinkle: opinions on Popups having two package.json files? https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Popups/+/master/.storybook/package.json - should libraryupgrader be recursively looking through the repo tree for this stuff or just deal with the root and ask maintainers to only have one package.json file?
[16:21:44] <James_F>	 legoktm: Hmm.
[16:22:39] <James_F>	 In cases like this, I think it's fine for LU to ignore it and make it the responsibility of the maintainers.
[16:26:27] <Krinkle>	 legoktm: yeah, ignore for now. It's not used in CI afaik, either.
[16:26:42] <Krinkle>	 Once it is, it'll be past the Node 10 upgrade, and thus not needing to be separate anymore.
[16:27:11] <legoktm>	 sounds good, thanks

I think we expect it to stay around for now.

Change 533628 had a related patch set uploaded (by Daimona Eaytoy; owner: Daimona Eaytoy):
[integration/config@master] Run PHP71+ jobs for mediawiki/tools/phan

https://gerrit.wikimedia.org/r/533628

So first, I note that ContentTranslation has seemed to gotten this working without any libup intervention, see e.g. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ContentTranslation/+/ec12768a718a8bccf070a70b86c9c42afe603362

I believe this works because of https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ContentTranslation/+/refs/heads/master/package.json where npm test runs both the main test and the sub test. Would doing something like that work for the Wikibase repos?

For actually implementing this in libup, recursing through subdirectories to run the npm ... commands should be straightforward. The main technical issue would be how to represent dependencies in libup's internal tracking database, especially if they're conflicting. For now, I think we could internally pretend each nested package.json is a separate repository with some fake name like mediawiki/extensions/Wikibase:client/data-bridge/package.json.

So first, I note that ContentTranslation has seemed to gotten this working without any libup intervention, see e.g. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/ContentTranslation/+/ec12768a718a8bccf070a70b86c9c42afe603362

The comment

I am not sure, why the sub package-lock.json was also updated.
It seems the sub package-lock.json is not whitelisted for libup to automerge.

and response

Yeah; I think this is rare enough that it's OK to do manually.

don't make it sound like that's a well known feature (if at all) and should be relied on. Would be great to see this functionality become a first class citizen (documentation et al.) before we build on top of it.

It should also be noted that we did not exactly invent the idea of monorepos and that - if anything - they can be expected to become more popular especially in a world of well-scoped micro frontends.

@Legoktm I hear you might have something running locally?
Anything we can help with to get this deployed? :)

@Legoktm we are still keen to make this work, did you have your WIP code somewhere? :)

An example is mediawiki/libs/metrics-platform.git which co host code for NodeJs, PHP, Swift and Java, each in their own directory having the relevant package manager file:

I'd love to see this as well, for GrowthExperiments where we have a documentation site with its own package.json (cc @Sgs )