Page MenuHomePhabricator

Host m3api documentation on doc.wikimedia.org
Closed, ResolvedPublic

Description

As part of migrating m3api away from GitHub in general (see the parent task), doc.wikimedia.org seems like a good new home for its documentation. I’ve been told (below this thread) that “volunteer” projects can be accepted there :)

The general documentation setup, which I’d like to keep, is that each release builds its own documentation, and then that documentation is mostly left untouched later. If I want to completely switch the documentation generator from JSDoc to something else in the future, I can do that and the past documentation won’t be affected – which means I won’t have to check if the doc comments of the past versions all render correctly.

More specifically, documentation is currently hosted via GitHub Pages. Each release checks out the gh-pages branch; adds a “This is an old version. The latest stable version is [link]vx.y.z” warning banner to the previous latest version, and also updates those banners in all the older versions; then builds the new docs. (GitHub Actions workflow, example commit.) There is also a small index.html file at the docroot to redirect to the latest version (this is also updated with each release). This design has several characteristics I consider desirable: when people look at the documentation and copy the link, that link will stay stable (because it always has the version number in the URL); there is a “latest” URL but only for the docroot (“latest” links for any particular class or function would break when that class or function is removed or renamed); when people load the documentation for an older release, they see a warning, but they’re not automatically redirected away.

The usual workflow for doc.w.o seems to be to just rsync the freshly built current docs over (either into a subdirectory named after the version, or directly into the project root if you only want latest documentation); the docs aren’t tracked in a separate Git repository or branch. However, to make something like the above process work (including updating the “latest version” banners in all the old docs), I think the easiest solution will be to have the docs in a Git branch after all, and do something pretty similar to the GitHub pages setup – and then to just rsync all the docs over at the end (and let rsync work out that e.g. /v1.0.0/styles/ didn’t change).

Event Timeline

This will probably take a few attempts to get right – would it be okay to first try it out with a temporary / experimental package (still published to the “real” doc.w.o), and to then remove that again from doc.wikimedia.org later?

In tmp-m3api-example (edit: and tmp-m3api), I now have a CI workflow that commits and updates documentation in the doc branch, and rsyncs it – to my own server, for now: see v0.0.17 and v0.0.18 (edit: better yet, v0.0.19 and v0.0.20). I think basically the same thing should also work for doc.wikimedia.org except with a different rsync command (and, if I read docpub correctly, with an rsync password rather than SSH key).

Just for the record here – I’m still working on other parts of T392290, but I’m pretty sure I’ve reached the limit of what I can do on this task on my own, so at some point I’ll need someone else’s help to unblock this (either let me know how to get rsync access to doc.w.o, or tell me that this isn’t going to happen after all and I need to find another place to publish the docs).

For now I’ve updated the CI templates to push the docs to GitHub, so the documentation can continue to be hosted on GitHub pages, unblocking the rest of the m3api migration. Maybe some day this task will happen, but after almost five months I really don’t feel like letting it block the migration any longer.

This will probably take a few attempts to get right – would it be okay to first try it out with a temporary / experimental package (still published to the “real” doc.w.o), and to then remove that again from doc.wikimedia.org later?

@Reedy said this is probably fine :)

Bikeshedding question: what should the URLs be?

(We might not have a choice here – docpub might insist on one or the other – but I’m still curious if anyone has a strong opinion either way ^^)

Trying this out in the new tmp-m3api-example repo, using the wip branch (current tip but I may force-push that away later) of the ci-templates.

It works \o/ \o/ \o/

https://doc.wikimedia.org/m3api/tmp-m3api-example/ (archived)

Next steps: clean up the ci-templates; apply the new version to all the existing packages; get the existing packages added to the allowlist; push out new release to do the initial doc publish; get the tmp package dropped from the allowlist again; ask someone™ to rm -rf m3api/tmp-m3api-example again from doc.wikimedia.org. Not necessarily in that order :)

Bikeshedding question: what should the URLs be?

(We might not have a choice here – docpub might insist on one or the other – but I’m still curious if anyone has a strong opinion either way ^^)

Opinion on fedi seemed to lean towards the latter option, so unless somebody objects, I’ll go ahead with that. (I’ll also make sure to mention it in the MR adding the other repos to the allowlist, for visibility.) Let’s first try out in the tmp repo if that’s possible.

Nope, not allowed. CI output (not kept forever because my volunteer account doesn’t have permission to do that apparently ^^):

14:36:49 ERROR Publishing location "tmp-m3api-example" is invalid for project "m3api/tmp-m3api-example"

Looking at the code suggests that the $PUB_LOCATION is only meant to allow putting the docs into further subdirectories of the default value; changing the prefix is not allowed. Let’s just go with the longer, slightly redundant URLs then.

lucaswerkmeister opened https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/35

Revert "Add repos/m3api/tmp-m3api-example to allowed_projects"

Alright, now I think we just need someone™ with the right permissions to effectively rm -r https://doc.wikimedia.org/m3api/tmp-m3api-example/. (Looking at httpd-doc.wikimedia.org.conf, I think it would be at /srv/docroot/org/wikimedia/doc/m3api/tmp-m3api-example/? On doc1004 and/or doc2003?) @bd808 or @jnuche can you do that? :)

@LucasWerkmeister I could find the docs at /srv/doc/m3api/tmp-m3api-example. The dir is in both hosts, it seems doc2003 is the primary and doc1004 is a sync from that.

Unfortunately I don't have the permissions to delete the files, let's see if @bd808 does. Otherwise I'm also pinging @Dzahn, I'm sure he'll have access

Unfortunately I don't have the permissions to delete the files, let's see if @bd808 does.

My contint-admins membership lets me ssh into those boxes, but there do not appear to be any sudoer rules that let me do anything useful like become the doc-uploader user. Looks like cleanup needs help from a full root.

Huh, thanks. @Ladsgroup could you perhaps rm -r /srv/doc/m3api/tmp-m3api-example on doc2003?

Sorry, got reminded of this now.

Checked if the same file was already rsynced to doc1004. It's clean there.

Also I was under the impression that the rsync upload already happens from CI and not from a manual user.

We could probably add a herald rule to tag doc.wikimedia.org with a team tag for visibility.

My contint-admins membership lets me ssh into those boxes, but there do not appear to be any sudoer rules that let me do anything useful like become the doc-uploader user. Looks like cleanup needs help from a full root.

The contint-admins group does have:

'ALL = (doc-uploader) NOPASSWD: ALL',

so any command as doc-uploader should actually work.

(sudo -u doc-uploader rm ...)

LucasWerkmeister claimed this task.

Also I was under the impression that the rsync upload already happens from CI and not from a manual user.

Yes, this was to clean up a CI upload I did under a temporary project name :)

Anyway, it’s gone now \o/

image.png (520×454 px, 36 KB)

Thanks everyone!

My contint-admins membership lets me ssh into those boxes, but there do not appear to be any sudoer rules that let me do anything useful like become the doc-uploader user. Looks like cleanup needs help from a full root.

The contint-admins group does have:

'ALL = (doc-uploader) NOPASSWD: ALL',

so any command as doc-uploader should actually work.

(sudo -u doc-uploader rm ...)

Thanks for pointing that out @Dzahn. I had somehow forgotten that there are sudoer roles inside the data.yaml configuration. I was only looking in the Puppet modules applied directly to the doc* hosts for rules. Your answer also led me to lookup how to get sudo itself to tell me what grants apply to my user. I'm not sure why I never figured out sudo --list exists before, but TIL.

Ah!:) sudo --list is nice indeed! Yes, ideally the sudoer rules are not supposed be in modules at all but all applied like this:

sudo-privs -> admin group

admin group -> puppet role

puppet role -> servers

So "grep -r contint-admins" in puppet/hieradata gets the roles the group is applied to (ci.yaml and doc.yaml); site.pp the roles applied to the hosts and the admin module has the actual sudo privs of that group.

I would have gone from id bd808 to tell me the groups to sudo cat /etc/sudoers.d/contint-admins but of course that itself requires root. So that's a nice point you added there that I will keep in mind.