Related:
T367322: Create a Maven package registry in Gitlab
T379187: Create new GitLab project: repos/wmf-packages

@brennen it'd be nice if we could use the repos/wmf-packages project for npm packages too, but this gitlab naming convention requirement seems to mean we can't? To use published @wikimedia scoped npm packages, (e.g. @wikimedia/eventgate), the packages must be published to a either a repos/wikimedia...or maybe a repo prefixed with repos/wikimedia, e.g. repos/wikimedia/packages ?

If repos/wikimedia/packages would work, I wonder if we could rename repos/wmf-packages to repos/wikimedia/packages?

@tchin do you think that would work?

In T384364#10527963, @Ottomata wrote:

Related:
T367322: Create a Maven package registry in Gitlab
T379187: Create new GitLab project: repos/wmf-packages

@brennen it'd be nice if we could use the repos/wmf-packages project for npm packages too, but this gitlab naming convention requirement seems to mean we can't? To use published @wikimedia scoped npm packages, (e.g. @wikimedia/eventgate), the packages must be published to a either a repos/wikimedia...or maybe a repo prefixed with repos/wikimedia, e.g. repos/wikimedia/packages ?

If repos/wikimedia/packages would work, I wonder if we could rename repos/wmf-packages to repos/wikimedia/packages?

@tchin do you think that would work?

npm naming convention docs relevant bits:

If you plan to install a package from a project or group, then you do not have to adhere to the naming convention.
[...]
If you plan to install a package from an instance, then you must name your package with a scope. [where] Project: https://gitlab.com/my-org/engineering-group/analytics → Scope: @my-org

Installing from an instance and using @wikimedia/eventgate would mean we'd need https://gitlab.wikimedia.org/wikimedia.

But is it possible to use a group or project scope for you all?

npm install docs relevant bits:

npm config set @scope:registry=https://<domain_name>/api/v4/projects/<project_id>/-/packages/npm/
npm install @scope/my-package

So that would be like:

$ # 2745 is project id for wmf-packages
$ npm config set @wikimedia:registry=https://gitlab.wikimedia.org/api/v4/projects/2745/packages/npm/
$ npm install @wikimedia/eventgate

Does that make sense? I could be misreading the docs here or misunderstanding what you're asking.

Ya makes sense! And we do that now.

But, users of packages have to know and configure which gitlab project id to use for which packages.

It would be nice if gitlab package registries could work more like other package registries: There is a single package registry url that the package manager can use to search for and install packages from.

Ideally, WMF would have a global package registry for all kinds of (library) packages we might want to publish, e.g. npm, jars, wheels, conda packages, docker images (we have this!!), etc.

Then, users could configure their package manager to use the global WMF registry, without also having to know which gitlab project the package was associated with.

Got it. Well. The "instance" scope seems fully bought-in to the github model of each org has a top-level; i.e., github.com/wikimedia, so it sounds like we need a gitlab.wikimedia.org/wikimedia with any npm packages we want to publish living under that namespace.

More questions for folks on this task:

• Who is using these packages, currently?
• What repos, specifically, would need to move there for this task?
• Are there permissions needed at the /wikimedia level (as far as you all are aware)?
• Does this affect repos/wmf-packages?
  • Note that if we moved repos/wmf-packages to, say, wikimedia/maven it would leave behind a 301 at repos/wmf-packages to redirect folks (at least from the web/git side)

Broader questions:

• How do we avoid confusion about what lives in wikimedia vs. what lives in repos?
• What's the criteria we can document about what npm packages get to publish there? If a third party who hosts with us wants to publish an npm package with a @wikimedia scope, can they? What do we do about future scopes (e.g., @wikimedia-de)?

Who is using these packages, currently?

Any team that uses service-runner, or hopefully better the new service-utils, so most production nodejs services, as well as any @wikimedia packages service-utils depends on.

Other nodejs libraries include
@wikimedia/url-get
@wikimedia/eventgate
@wikimedia/node-rdkafka-factory, etc.
@wikimedia/node-rdkafka-prometheus
@wikimedia/kafkasse
@wikimedia/jsonschema-tools (used in e.g. schemas-event-primary and schemas-event-secondary

@tchin made a really nice diagram of some nodejs dependencies for a demo he did recently:

To expand the scope a bit beyond npm, it'd be nice to have a global registry for other things too.

• T367315: [Epic] Replace Archiva with Gitlab artifact repositories
• T367406: Migrate Python projects that depend on Archiva for deployment

There are several java and python libraries that are used in projects, e.g.

wikimedia-event-utilties (java) <- eventutilities-python <- mediawiki-event-enrichment

Not all of these are yet in gitlab or off of archiva.

What repos, specifically, would need to move there for this task?

I don't think any repos themselves need to move. They just need to be configured (with gitlab CI templates like the ones in workflow_utils ) to publish to the global package registry. Related: T382430: Create a GitLab CI/CD Component project for WMF CI/CD templates and components

If the package is for more than WMF usage, then probably the main/default language registries are more appropriate. E.g. where should eslint-config-wikimedia be published to?

Does this affect repos/wmf-packages?

Probably. This should be moved to something like /wikimedia/packages or something?

moved repos/wmf-packages to, say, wikimedia/maven

Gitlab supports multiple registry types in one project. So no need to have multiple projects for each kind of registry.

How do we avoid confusion about what lives in wikimedia vs. what lives in repos?

Code can still live in /repos/<project>. Packages can be published to /wikimedia/packages

What's the criteria we can document about what npm packages get to publish there? If a third party who hosts with us wants to publish an npm package with a @wikimedia scope, can they? What do we do about future scopes (e.g., @wikimedia-de)?

Hm, good question.

In general, I think any package that is intended to be used as a library in more than one WMF project should probably be published to a WMF global registry.

@wikimedia-de could probably just use the @wikimedia scope? Not sure!