The .pipeline files are used to create container images for running in Wikimedia production or anywhere else that may choose to run Shellbox as a container. The CI tests for the project are also part of the PipelineLib integration.

I am sure it would be possible to design a replacement such that we have a second repository which manages the container images something like a typical Debian packaging setup where the code being packaged is imported separately. What I'm not sure of is the effort that will require and who would do the work. It appears that PipelineLib has been in use via 2f4a7de since very early in the project's life.

If there is a specific issue with the syntaxhighlight/ directory I added for T320848: Install pygments in Shellbox container with pip, not a Debian package causing problems with reusing the library I can make some time to try to remedy those issues once they are explained.

In T357949#9565402, @bd808 wrote:

The .pipeline files are used to create container images for running in Wikimedia production or anywhere else that may choose to run Shellbox as a container.

You're saying that anyone anywhere who wants to run Shellbox in a container, whether they are using MediaWiki or not, should use the exact same configuration as WMF production, down to the PHP version, number of FPM workers, list of fonts to install, etc.?

If someone wanted to use Shellbox for some other thing, would you expect every detail of every container they build to be submitted as a patch and included in .pipeline?

I am sure it would be possible to design a replacement such that we have a second repository which manages the container images something like a typical Debian packaging setup where the code being packaged is imported separately.

I think doing so would provide a good model for third-party installations to follow, and would establish a convention for similar services going forwards.

What I'm not sure of is the effort that will require and who would do the work. It appears that PipelineLib has been in use via 2f4a7de since very early in the project's life.

It was set up by ServiceOps and they are still maintaining it.

In T357949#9565748, @tstarling wrote:

You're saying that anyone anywhere who wants to run Shellbox in a container, whether they are using MediaWiki or not, should use the exact same configuration as WMF production, down to the PHP version, number of FPM workers, list of fonts to install, etc.?

I'm saying that https://www.mediawiki.org/wiki/Shellbox#Pre-built_containers references these containers and that they are usable in a Wikimedia production matching manner.

@tstarling just removing the .pipelinelib directory isn't an option, if we want to keep using shellbox in production. I don't see a good solution for than other than maintaining separate branches and backporting changes to the main branch to a wmf branch, which would come with its own inconveniences of course.

Do you have another, better solution in mind?

In T357949#9565748, @tstarling wrote:

In T357949#9565402, @bd808 wrote:

The .pipeline files are used to create container images for running in Wikimedia production or anywhere else that may choose to run Shellbox as a container.

You're saying that anyone anywhere who wants to run Shellbox in a container, whether they are using MediaWiki or not, should use the exact same configuration as WMF production, down to the PHP version, number of FPM workers, list of fonts to install, etc.?

No, but they could use the container we provide as a base and build on top of that, or roll out their own.

If someone wanted to use Shellbox for some other thing, would you expect every detail of every container they build to be submitted as a patch and included in .pipeline?

No, I'd expect them to either:

• Use one of our containers as a base, build a dependent image of their own
• Build their own container from scratch

I should add, nowadays it's not uncommon at all for software to include ancillary files for building specific docker images or pipelines for CI in the environment they're developed in.

In T357949#9566576, @Joe wrote:

I don't see a good solution for than other than maintaining separate branches and backporting changes to the main branch to a wmf branch, which would come with its own inconveniences of course.

I think something like a mediawiki/libs/Shellbox/deploy repository that pulls in Shellbox via composer (or just git)? during build and other Wikimedia-needed files wouldn't be too much work to maintain.

Before we move into finding solutions, I'd like to understand better what is the goal we want to accomplish:

• If the goal is to make what we upload to packagist cleaner, composer.json has a way to do it (archive.exclude, IIRC?)
• If we prefer not to have pygmentize in the repository (why?), we can adapt the image build process to fetch it remotely

If the problem is just that those files are wmf specific, then I don't see the problem with having them in the repository but not the artifacts we produce.