Page MenuHomePhabricator

NPM packaging strategy for Codex
Closed, ResolvedPublic

Description

Right now the Codex repo has four workspaces that each build their own build products, but none of them build something that we could publish to NPM. The main approaches I think we could take are:

  1. Publish each workspace as a separate package
    • This would require renaming the workspaces to their equivalent NPM package names, e.g. codex-icons instead of icons, codex-tokens instead of design-tokens and codex-components (or codex?) instead of vue-components
    • End users would have to import both the components and the icons package for typical usage, which is not very user-friendly. End users that need tokens would also need to import a third package.
    • We would be able to deliver separate CommonJS/UMD builds for components and icons without having to do anything weird in our Vite config
    • Publishing would be more laborious, because there would be more packages to keep track of
  2. Publish everything as one big package
    • This would not require renaming any existing workspaces. We'd publish from a new, separate workspace.
    • End users would have a more user-friendly experience of being able to use one NPM package that gives them everything they need (components, icons and tokens)
    • Delivering split CommonJS/UMD builds could be trickier: they would probably require more configuration, and we would likely have to come up with unique global variable names for each of them
  3. Combine 1+2: publish each package separately, but also publish a wrapper package that groups them (similar to what Vue does)
    • End users would still be able to use one package for everything
    • CJS/UMD end users who need separate builds could get them from the subpackages; we wouldn't need much additional configuration to make this work
    • We would still have to rename our workspaces to match the subpackage names
    • We would have to keep track of even more NPM packages that we publish (four instead of three)

Event Timeline

I initially favored #3, but now I think I might favor #2. Extra config will be needed for the split CJS/UMD builds, but I think that extra complexity would be worth the simplicity of only publishing one NPM package. We could perhaps also bypass the need for extra config by copying the CJS/UMD builds generated by the individual workspaces into the released package.

Let's work backwards from the use-cases that we think we're going to need to support and see where that leaves us:

  • Usage with ResourceLoader
    • We need CJS (for usage in package files) and IIFE (for traditional usage); perhaps a single UMD bundle that covers both is best
    • Bundles intended for use in RL should be as granular as possible to reduce the amount of unnecessary code that gets loaded
    • Different extensions may load different parts of Codex on the same page at different times using mw.loader.using; the IIFE or UMD bundles should probably add things globally as sub-properties of window.Codex – window.Codex.icons, window.Codex.vueComponents, etc. This is similar to how OOJS and OOUI are delivered now.
  • Usage with a build step
    • ES modules preferred here, one big bundle with named exports should be fine. This allows for tree-shaking.

@Catrope I think I agree that we want option 2 (publish a single big package) but that we'll have to do some manual configuration to organize the CJS/UMD bundles.

One other thought – I think that the typeahead search widget (which composes several components into a mini-app basically) should be treated as yet another workspace and bundled separately (with appropriate externalized peer dependencies). It's something of a special case and by treating it as a separate workspace (with its own vite config and dedicated bundle files) we can ensure it stays fully optimized.

One other thought – I think that the typeahead search widget (which composes several components into a mini-app basically) should be treated as yet another workspace and bundled separately (with appropriate externalized peer dependencies). It's something of a special case and by treating it as a separate workspace (with its own vite config and dedicated bundle files) we can ensure it stays fully optimized.

Hmm, I would be OK with that, but I would want to develop a more complete justification for why it's so special and separate. Having a separate UMD build for it is one thing, but having it live in an entirely separate workspace is another. Are we saying it's not really part of the Codex library of base components because it's MediaWiki-specific, or otherwise a very specific application?

Are we saying it's not really part of the Codex library of base components because it's MediaWiki-specific, or otherwise a very specific application?

It's intended to be used in a very specific way at any rate (based on the needs of the Vector skin and the Web team generally). There is the question of whether we build something that can be plugged in to any API or has special support for various Wikipedias, Wikidata, etc. We might also need to include some code to support MediaWiki-specific instrumentation. So we may want to decouple this feature from the rest of the component library.

If we don't actually need to do any of the things I listed then maybe there is not a need for this.

#2 is a no-go for Design Systems approach from my current understanding, as it would disable combining tokens and icons into a different technical implementation – think for example Apps using only tokens or a React implementation of the Design System, where tokens and icons are essential.

Is it necessary to have either codex- or vue- presiding components?
Do we think we might have different implementations in the same repo? Isn't components sufficient for the directory?

Is it necessary to have either codex- or vue- presiding components?
Do we think we might have different implementations in the same repo? Isn't components sufficient for the directory?

Yes, I think that would be fine.

#2 is a no-go for Design Systems approach from my current understanding, as it would disable combining tokens and icons into a different technical implementation – think for example Apps using only tokens or a React implementation of the Design System, where tokens and icons are essential.

I don't think it would break that ability, because you could use only parts of the package. But it would certainly be more convenient and hygenic to make them available as separate packages.

It's intended to be used in a very specific way at any rate (based on the needs of the Vector skin and the Web team generally).

In future, we also want to use this in the Minerva skin and I could see us using it for other purposes e.g. searching categories, searching reading lists (https://www.mediawiki.org/wiki/Extension:ReadingLists) and on the long long term as a replacement for the OO.ui.SearchWidget

The problem is right now it is very specific to the use case of searching pages. I'd love this to be stripped away from the library and moved into Vector/core.

Having Codex in NPM would also be useful for pages that live off MediaWiki like Wikistats, which we will explore here: T298816. Let's keep this task open as a prototype for getting NPM into Codex and we can create new tickets when it comes to project integrations. We can move on this task once we have a clear first use-case for it (will follow up here).

STH renamed this task from Figure out packaging strategy for Codex to NPM packaging strategy for Codex.Jan 13 2022, 5:53 PM

It seems to me like the immediate need is to have an initial release of the vue-components and icons packages under whatever names make sense. This would enable other teams to start alpha-testing these components and providing feedback.

The tokens package could be released a little bit later if we are still putting the basic organization in place. Eventually other products will probably want to use these for developing custom components in a way that follows the design system guidelines, but for now the tokens package could remain an internal dependency of Codex components.

The vitepress package (documentation site) may never need to be published in this way at all; we publish a static site instead.

In terms of approaches, I'd be in favor of Option 1:

  • Publish vue-components as @wikimedia/codex
  • Publish icons as @wikimedia/codex-icons
  • Publish tokens when ready as @wikimedia/codex-tokens (if these are truly implementation-agnostic we could also just publish these as @wikimedia/design-tokens or something similar)

I don't know that we need a single package for everything. Within MediaWiki, most users are going to want the Icons package handled separately so that they can only load necessary icons via RL callback params (example).

Outside of MW, adding a separate line to import { someIcon } from '@wikimedia/codex-icons seems fine. But I'd be concerned that a single package containing everything (all icons, components, etc) and distributed in UMD format would be difficult to tree-shake; I'd prefer to discourage such "all-or-nothing" usage if possible.

One related point to consider is what kind of versioning policy we want to follow in NPM. I assume we'll want to follow Semver for this, right?

For initial alpha (and maybe also beta) releases I'd be in favor of designating them as such, per the semver docs:

A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphens [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1

If we followed this, we could start with 0.1.0-alpha.1 and increment from there if we do any necessary "sub-releases" (for example, adding a new component to the pre-release copy so a pilot project can use it).

I like @egardner's proposal in T294993#7669176. Here's my more detailed version of it:

  • Rename packages/icons to packages/codex-icons and publish it to NPM as @wikimedia/codex-icons
  • Rename packages/design-tokens to packages/codex-tokens and publish it to NPM as @wikimedia/codex-tokens
  • Rename packages/vue-components to packages/codex and publish it to NPM as @wikimedia/codex
    • This package does not need to depend on the icons or tokens packages, because it doesn't use them at runtime, only at build time
  • Rename packages/vitepress to packages/codex-docs. Give it an internal name of @wikimedia/codex-docs, but don't publish it to NPM. (I'd like to rename this one for consistency, and so that we don't have an internal package and an external dependency with the same name. I'm honestly shocked this hasn't already caused problems.)

Alternatively, if we want to provide a single package for everything, we could instead:

  • Rename packages/vue-components to packages/codex-components, and publish it to NPM as @wikimedia/codex-components
  • Create packages/codex which just exports everything from the components and icons packages, and publish it to NPM as @wikimedia/codex.

This would enable code like import { CdxTextInput, cdxIconAlert } from '@wikimedia/codex';, but I don't know how worthwhile that would be, you could just import them from separate packages instead.

+1 to the separate package strategy and the names proposed by Eric and Roan

Change 759630 had a related patch set uploaded (by Catrope; author: Catrope):

[design/codex@main] Rename packages so they're suitable for publication to NPM

https://gerrit.wikimedia.org/r/759630

Change 759630 merged by jenkins-bot:

[design/codex@main] Rename packages so they're suitable for publication to NPM

https://gerrit.wikimedia.org/r/759630

Late to the party, is codex-tokens self-explaining? Would codex-design-tokens be too verbose? (I got some doubts with the shortening after seeing the merged patch)

Late to the party, is codex-tokens self-explaining? Would codex-design-tokens be too verbose? (I got some doubts with the shortening after seeing the merged patch)

I think codex-tokens is okay, the whole project is a design system / component library so I think it is reasonable to assume that "tokens" here means "design tokens", as opposed to say tokens of the non-fungible variety...

Change 760679 had a related patch set uploaded (by VolkerE; author: VolkerE):

[design/codex@main] build: Rename 'codex-tokens' to 'codex-design-tokens'

https://gerrit.wikimedia.org/r/760679

Change 760679 merged by jenkins-bot:

[design/codex@main] build: Rename 'codex-tokens' to 'codex-design-tokens'

https://gerrit.wikimedia.org/r/760679

This is complete! We've chosen to publish each workspace as a separate package (option 1 from above).

Here are the components: https://www.npmjs.com/package/@wikimedia/codex
Here are the icons: https://www.npmjs.com/package/@wikimedia/codex-icons

Tokens will be published when ready, prior to 1.0 release of Codex.

Currently the codex packages have "main" and "module" entries in their package.json:

"main": "dist/codex.umd.js",
"module": "dist/codex.es.js"

I don't know what's a valid mechanism to import components from the es module in a non-MW environment. For instance when trying to build a VitePress project which imports Codex components as import { CdxButton } from '@wikimedia/codex'; the following error is yield:

import { CdxDialog, CdxButton } from "@wikimedia/codex";
                    ^^^^^^^^^
SyntaxError: Named export 'CdxButton' not found. The requested module '@wikimedia/codex' is a CommonJS module, which may not support all module.exports as named exports.
CommonJS modules can always be imported via the default export, for example using:

import pkg from '@wikimedia/codex';
const { CdxDialog, CdxButton } = pkg;

Have you considered using package.json "exports" entry? I was able to import the es Codex components and icons by adding an "exports" entry as follows:

"main": "dist/codex.umd.js",
"module": "dist/codex.es.mjs",
"exports": {
  "import": "./dist/codex.es.mjs",
  "require": "./dist/codex.umd.js"
}

(Note in my tests I had to use the .mjs extension to get it working, not sure why). Before that I had also tried "type":"module" in the project package.json but didn't work. I think for that we would need to change the "main" entry. Is there a preferred mechanism to import from the es module that I'm missing?