@Joe and @Reedy, I would like your perspectives on this topic. What pros/cons stand out the most for each of you about the idea of hosting in the main network versus Toolforge? Is there a thing either of you can point to that makes one or the other the most obvious choice?

In a discussion with @srishakatux, we ended up leaning pretty heavily towards the idea of using Toolforge as staging system for user acceptance testing and the Wikimedia Foundation's production Kubernetes cluster plus support services for the user facing site. The benefits of existing log aggregation and monitoring systems we can leverage in production feel like they outweigh the increased friction for getting deploy access and coordinating with other teams for using their shared services (mariadb, elasticsearch, etc). Having a staging environment in Toolforge would still give us an opportunity to get technical volunteers involved in deployments and maintenance and then possibly sponsor them for production deployment rights if there is interest and need later.

I would still like to hear from @Joe and @Reedy for any additional points they might be able to make for or against these options. Or even better to hear a 4th option that makes my 3 initial ideas look sad. :)

In T261017#6434916, @bd808 wrote:

In a discussion with @srishakatux, we ended up leaning pretty heavily towards the idea of using Toolforge as staging system for user acceptance testing and the Wikimedia Foundation's production Kubernetes cluster plus support services for the user facing site. The benefits of existing log aggregation and monitoring systems we can leverage in production feel like they outweigh the increased friction for getting deploy access and coordinating with other teams for using their shared services (mariadb, elasticsearch, etc). Having a staging environment in Toolforge would still give us an opportunity to get technical volunteers involved in deployments and maintenance and then possibly sponsor them for production deployment rights if there is interest and need later.

I would still like to hear from @Joe and @Reedy for any additional points they might be able to make for or against these options. Or even better to hear a 4th option that makes my 3 initial ideas look sad. :)

Apologies for taking a long time to respond, but you assigned the task to me at the start of my unforeseen leave of absence that ended yesterday :)

So, let's exclude option 3) without further discussion for the reson you cited.

Option 1) is the most sound from a technical POV, IMHO. Python support for the pipeline/blubber already exists, and thus deploying this software should be mostly matter of writing an appropriate .pipeline directory in the repository, and then add a suitable helm chart (which should be mostly trivial as well).
This would not solve the social problems you cite here - specifically it would require a production deployment access to allow deploying / restarting the software. Also it would not resolve the need to ask for storage resources (which should be picked up with the data persistence team and other teams).

Option 2) is ok technically but has the downside to have the directory being part of the PaaS it's being a directory for, in addition to the other issues you outlined. I usually advice against doing that, as it would become unusable for hosting something like e.g. a toolsforge external status page, which you might desire to do at some point.

If we want to go with option 1) we need to plan for it, specifically:

• Serviceops will need to allocate some time to assist with the deployment and reserving a couple redis instances for the celery backend
• We will need to request some space on the elasticsearch production clusters for this tool. Not sure how we would proceed there, so maybe we need to involve the Search Platform team early in the process
• We will need to request some space on the mariadb misc cluster to the data persistence team

None of the above asks is weird or unusual, and nothing (including running a python app on kubernetes) will be uncharted territory come next year. At this point, I'd summarize the above as follows:

Host in production
Pros:

• More dedicated resources possible
• CDN
• Standard for production-grade services, meaning it will benefit from the evolution of the deployment pipeline.
• Staging
• Multi-dc
• SRE support available
• Independent of toolforge

Cons:

• More intra-team dependencies for the setup
• Harder to delegate trust
• More work upfront

I have one major objection to option 2 (which is otherwise viable): it would mean hosting a tools directory on the same platform as the tools it's listing, which means any issue of the platform would reflect. Apart from that, the disadvantages would be having lesser available resources, the absence of SRE support (but AIUI WMCS could provide SRE support on toolsforge?), and probably overall a less "standard" installation compared to the rest of our production stack. On the other hand, it would allow an easier volunteer involvement in maintaining the tool, and less steep upfront investment in the initial setup.

In the end, I think it's a choice between "more reliability" versus "easier setup/community involvement in maintenance", and I guess it's up to you to make the call. I have no major objections to going with either option :)