Page MenuHomePhabricator

Deploy current main branch's docs site to doc.wikimedia.org
Closed, ResolvedPublic

Description

We want a live version of the VitePress site up on doc.wikimedia.org.

Event Timeline

I tried writing a JJB patch for this myself, but it looks tricky, since the directory that we want to publish isn't the docs/ directory in the root of the repo, but the packages/vitepress/docs/.vitepress/dist directory.

Change 735059 had a related patch set uploaded (by Catrope; author: Catrope):

[design/codex@main] build: Reorganize Vitepress Vite config and package.json

https://gerrit.wikimedia.org/r/735059

Change 736101 had a related patch set uploaded (by Catrope; author: Catrope):

[integration/config@master] Deploy Codex docs to doc.wikimedia.org

https://gerrit.wikimedia.org/r/736101

Catrope moved this task from Inbox to Infrastructure on the Codex board.

Something similar to what you seem to aim for is done when e.g. publishing the data-bridge storybook to https://doc.wikimedia.org/#Wikibase. The repo root is in control and delegates to different sub folders to build their individual documentations.

The generic job generic-node14-docs-docker-publish indeed assume the docs are generated under the docs directory. Someone must had the use case of running an arbitrary docker image to generate doc and get it from a different dir. An example is:

jjb/integration.yaml
- project:
    name: 'integration-pipelinelib'
    jobs:
     - '{name}-docker-doc-publish':
         image: 'docker-registry.wikimedia.org/releng/gradle:0.3.0-s4'
         args: 'groovydoc'
         docsrc: 'src/build/docs/groovydoc'
         docdest: 'pipelinelib'

Which creates a job integration-pipelinelib-docker-doc-publish which runs docker run docker-registry.wikimedia.org/releng/gradle:0.3.0-s4 groovydoc

It would publish the content from ./build/docs/groovydoc to https://doc.wikimedia.org/pipelinelib.

I guess you could use the same pattern:

- project:
    name: 'codex'
    jobs:
     - '{name}-docker-doc-publish':
         image: 'docker-registry.wikimedia.org/releng/node14-test:0.0.2-s2'
         args: 'doc'
         docsrc: 'packages/vitepress/docs/.vitepress/dist'
         docdest: 'codex'

Maybe add it to a new jjb/design.yaml file :)

I have commented on the change https://gerrit.wikimedia.org/r/c/integration/config/+/736101

Edit: and I forgot we have some documentation about it at https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation

Change 735059 merged by jenkins-bot:

[design/codex@main] build: Reorganize Vitepress Vite config and package.json

https://gerrit.wikimedia.org/r/735059

Change 736101 merged by jenkins-bot:

[integration/config@master] Deploy Codex docs to doc.wikimedia.org

https://gerrit.wikimedia.org/r/736101

Change 736996 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Fix Codec docsrc

https://gerrit.wikimedia.org/r/736996

Change 736996 merged by jenkins-bot:

[integration/config@master] Fix Codec docsrc

https://gerrit.wikimedia.org/r/736996

I have manually triggered the postmerge pipeline against the latest merged change in the repository https://gerrit.wikimedia.org/r/c/design/codex/+/736884 using:

contint2001$ zuul enqueue --trigger gerrit --pipeline postmerge --project design/codex --change 736884,2

There was a tiny mistake the doc source parameter which I have adjusted. The content of packages/vitepress/docs/.vitepress/dist has been successfully at https://doc.wikimedia.org/codex/main/

The site is broken though I get some 404:

https://doc.wikimedia.org/assets/app.2f091619.js
https://doc.wikimedia.org/assets/style.78cfa091.css

And from the console (under Firefox):

Loading module from “https://doc.wikimedia.org/assets/app.2f091619.js” was blocked because of a disallowed MIME type (“text/html”).

If I run npm run doc locally, I do have a ./packages/vitepress/docs/.vitepress/dist/assets. The directory has been copied to doc1001.eqiad.wmnet under /srv/doc:

doc1001.eqiad.wmnet
$ find /srv/doc/codex/main -name assets
/srv/doc/codex/main/assets

But the generated documentation uses absolute paths: <a href="/assets/...">.

Change 737151 had a related patch set uploaded (by Catrope; author: Catrope):

[design/codex@main] docs: Update base URL

https://gerrit.wikimedia.org/r/737151

Change 737151 merged by jenkins-bot:

[design/codex@main] docs: Update base URL

https://gerrit.wikimedia.org/r/737151

Amazing @Catrope , I suspected a base url had to be filed in but could not find the setting. From the patch:

packages/vitepress/docs/.vitepress/config.js
module.exports = {
  base: '/codex/main',

Whenever a new release is tagged (eg: v1.0) or a new branch is cut (release_1), CI will publish to /codex/v1.0 or /codex/release_1 and there resulting doc will retrieves its assets from the main path which will surely break things :(

The doc at https://vitepress.vuejs.org/config/basics.html#base states It should always start and end with a slash :-\

I have tried with base: './' and it does not complain, but in the rendered documentation the href all have ./assets/xxxx.... regardless of where they are in the directory tree so that would break.

I found an upstream issue: https://github.com/vuejs/vuepress/issues/796

base can't be hard-coded in the repository since it depends of the current branch / tag (master, 2.7.0, 2.8.0, etc...)

Any way to make the GitHub link instead link to our Gerrit (or, soon, GitLab) instance?

Whenever a new release is tagged (eg: v1.0) or a new branch is cut (release_1), CI will publish to /codex/v1.0 or /codex/release_1 and there resulting doc will retrieves its assets from the main path which will surely break things :(

Yes, we'll have to do this manually in the release branch before tagging, similar to what we do for $wgVersion in MediaWiki. That's annoying and it's a manual process that's easy to forget, but we can do it. We could also try to detect this programmatically somehow (and get the version number from package.json). Ideally, though, I would like for upstream to fix this and use relative paths to get to the assets directory.

Could we consider an alternate solution where we use Netlify to deploy the site to codex.wikimedia.org? In my experience, setting up Netlify to automatically deploy the main branch when it's updated is extremely straightforward.

As far as coverage on doc.wikimedia.org, we could add an item for Codex to the homepage of doc.wikimedia.org similar to the item that exists for the Design Style Guide, which lives on design.wikimedia.org.

I don't think Netlify is self-hostable, is it? So I'm not sure if we could host it on a *.wikimedia.org domain directly (unless someone else has already figured that out). But I suppose we could host it on Netlify and have codex.wikimedia.org redirect to that.

Not only is netlify self hostable, it directly connects to a github repository and gives you a myriad of deploy tools, including an automatic pull request preview, A/B testing with two or more branches, deploy out of a specific branch, etc.

I understand the desire to stay in Gerrit in general but honestly, since this is a standalone static site that doesn't really use the "standard" CI given within our deploy stack, I highly encourage you to look into the much more straight forward approach of putting it on github and auto connecting it with Netlify deploy.

The usual question we encounter is privacy, but in this case, there's no PII at all, and it's a static site that doesn't have anything dynamic or user revealing, so I'd offer that point may be inconsequential.

Hosting in github with an auto preview/CI is also a much easier barrier of entry to any other developers who want to contribute to the documentation, and we have plenty of precedent for a standalone "unique" tools that are hosted in github.

I will even venture to suggest I'm not entirely sure what benefits you are getting from specifically the static documentation site being hosted in Gerrit at all? (I can see the benefit for codex itself, but the standalone vitepress instance, less so) especially given the extreme lengths netlify has gone through to make deploy tools exactly for this type of tool.

I would just like to add that nowdays the options are not limited to "Gerrit or Github" but there is also Gitlab. Anything moving from _either_ Gerrit or Github over to Gitlab would probably be a good thing at this point.