I think our biggest challenge here will be either figuring out how to export docs built inside a pipelinelib container out to the Jenkins runner node or building the docs outside of a container.

The technically "simple" method of getting the docs out of the container would be to mount a local directory on the runner node into the container at runtime. I don't know that pipelinelib actually supports that, but I did find something at https://wikitech.wikimedia.org/wiki/PipelineLib/Reference#Copy:

archive
Optional: Whether to archive file from destination in Jenkins, available at the project's lastSuccessfulBuild url found at: integration.wikimedia.org/ci/job/{pipeline-name}/lastSuccessfulBuild/
This is most useful to pipeline users who need to generate small artifacts during a post-merge build process and later use/download those artifacts remotely.

In T305914#7848631, @bd808 wrote:

I think our biggest challenge here will be either figuring out how to export docs built inside a pipelinelib container out to the Jenkins runner node or building the docs outside of a container.

How have other projects that are published on doc.wikimedia.org solved this? Or is Toolhub's setup unique?

In T305914#7848664, @Slst2020 wrote:

How have other projects that are published on doc.wikimedia.org solved this? Or is Toolhub's setup unique?

Toolhub is doing all of it's testing using PipelineLib which makes it not exactly unique, but uncommon. Most projects are using one or more of the common entry points and templated jobs to create their test environment and execute it directly on a Jenkins runner node. Toolhub instead uses PipelineLib to build a Docker container as the test environment and then define what to do in that environment to test the application.

In T305914#7848631, @bd808 wrote:

The technically "simple" method of getting the docs out of the container would be to mount a local directory on the runner node into the container at runtime. I don't know that pipelinelib actually supports that, but I did find something at https://wikitech.wikimedia.org/wiki/PipelineLib/Reference#Copy:

archive
Optional: Whether to archive file from destination in Jenkins, available at the project's lastSuccessfulBuild url found at: integration.wikimedia.org/ci/job/{pipeline-name}/lastSuccessfulBuild/
This is most useful to pipeline users who need to generate small artifacts during a post-merge build process and later use/download those artifacts remotely.

So basically, we would have to specify a new variant for building the docs in blubber.yaml, then add a doc pipeline to config.yaml with the copy archive arg set to true, and the docs would be downloadable from a specific url after a successfully completed job? And then make a bot that grabs them and pushes them (how?) to doc.wikimedia.org?

@hashar, would you be able to help us figure out the next steps?

@dduvall @jeena do we have past experience with a PipelineLib enabled repository publishing artifacts to doc.wikimedia.org ? I couldn't find anything at a quick glance.

There is some outdated documentation at https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation . The rough idea is that on a tag or post merge we trigger a freestyle Jenkins job which runs an image to generate the assets then the job invokes a builder which takes:

• a source path: relative to the job workspace, thus the container should had it mounted when generating the doc
• a destination path to doc.wikimedia.org

All that logic is intended for FreeStyle jobs and it is not reusable in the PipelineLib jobs.

In T305914#7903277, @hashar wrote:

All that logic is intended for FreeStyle jobs and it is not reusable in the PipelineLib jobs.

There is an undocumented action that PipelineLib supports called trigger that will trigger a downstream job (sync or async). (I've just updated the documentation.) Theoretically you could trigger the publish-to-doc job but that job needs a number of parameters from the upstream job to be able to rsync the generated files from the upstream workspace on the (possibly remote) agent host.

So maybe we could do it from PipelineLib. I'm not sure. Sorry, that's not a very satisfying answer. :)

@Slst2020 I'm up for hacking on it together if you'd like to propose a day/time (just not next week as we're doing a sprint). But please feel free to calendar stalk me.

Another option I just thought of: Since we do indeed support archiving files from a stage's container into Jenkins's default artifact store via copy.archive, maybe we could write a more simple downstream doc publishing job that grabs an arbitrary documentation archive from Jenkins over HTTP instead of trying to rsync it from a previous agent host/workspace. Seems a lot more sane in general and it could work well in combination with trigger to kick off the simplified publishing job.

At any rate, I'm available to help figure this out.

In T305914#7910841, @dduvall wrote:

@Slst2020 I'm up for hacking on it together if you'd like to propose a day/time (just not next week as we're doing a sprint). But please feel free to calendar stalk me.

Sounds great to me, thank you for proposing! A calendar invite is coming :)