The hostPath volume option implies either having git installed somewhere in the Pod, or shell access to the kubernetes node(s) hosting the Pod.

I'm also not a fan of having node affinity again, we just got rid of it for kask, and the fact that it doesn't actually guarantee the Pod will run there would cause failures because of the hostPath being undefined. That's true for both this task and T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).

I wonder if the use cases for both aren't close enough that they could be covered by the same tool that would:

• Start a Pod on demand, possibly on the staging cluster
• This Pod would run an initContainer toolbox with a shell, git, vim, other debugging tools?
• Take SHA1 revisions for the different code repos and auto checkout?
• Return the suitable host:port to use in XWD. The ATS lua code for the XWD header actually supports using an arbitrary backend, the extension doesn't but that can probably be added. The same host:port can be fed to the parsoid rt-testing script.
• Attach to the shell inside the initContainer for editing files, checking out a code revision, etc.

This obviously is full of holes and very handwavy, but I'd rather we support one sandbox environment that allows for multiple use-cases than multiple snowflake deployments with node affinities, puppet definitions, etc. if we can do everything within kubernetes.

Thanks, @Clement_Goubert!

In T386246#10677776, @Clement_Goubert wrote:

The hostPath volume option implies either having git installed somewhere in the Pod, or shell access to the kubernetes node(s) hosting the Pod.

I'm also not a fan of having node affinity again, we just got rid of it for kask, and the fact that it doesn't actually guarantee the Pod will run there would cause failures because of the hostPath being undefined. That's true for both this task and T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).

Indeed, a hostPath-based solution for parsoidtest is going to introduce exactly these complexities.

The sketch in T386246#10676755 isn't intended as a proposal to go this route - it's meant as an outline of what a solution of similar implementation complexity to the init-container approach would look like. Personally, I still tend to prefer the latter, chiefly because it constraints special-casing to the service itself rather than the environment.

I wonder if the use cases for both aren't close enough that they could be covered by the same tool that would: [...]

Implementation details aside, as I'm sure there are some interesting bits in there, I really like this concept as a more general solution for mutable sandbox environments, particularly for mw-experimental.

I'm certainly in favor of having a shared solution between mw-experimental and parsoidtest, or at least one that is sufficiently similarly shaped that there is a natural on-ramp to a shared solution.

Particularly given the short timeline on which we want to deliver a k8s-based parsoidtest, I think reaching for a fairly basic init-container approach like that sketched out in T386246#10572432 shouldn't stand in the way of converging the two when the more general solution lands.

And if the more general solution also ends up using an init-container approach for "self-contained environment setup" then piloting this with parsoidtest will probably prove useful.

Finally, I wanted to follow up on the last of the three options mentioned in the task description: a custom image build.

At a high-level, we would introduce a new image that is based on the latest mediawiki-multiversion image, where the desired parsoid commit ref is supplied by a build-arg and cloned during build. This would be quite similar to the way dumps is brought into the cli image built by make-container-image during scap deployments.

The primary benefit of this approach is that, like the hostPath option, gerrit is not in the critical path for pod startup. There are a couple of tricky questions to sort out, though:

Should this be a new image-kind directly supported in make-container-image and scap?

That would certainly make for the simplest implementation, since all of the necessary infrastructure for building, appropriately labeling, and pushing the image is already there. However, it would require a scap deployment in order to change the commit-ref-to-test. That carries a lot of baggage for such a narrowly scoped change, so I think the answer is no.

How would we manage builds that are external to make-container-image and scap?

Suppose this is managed by an external tool. While the build itself is straightforward, the tricky parts are now the book-keeping that would otherwise have been handled for us. One prime example is persisting built-image labels somewhere for consumption by helmfile - e.g., we don't want to introduce a non-scap writer to /etc/helmfile-defaults/mediawiki/releases.

Managing builds externally also commits to making the process asynchronous - i.e., scap updates helmfile-defaults, and at some point in the future we build a new image based on that (likely on a timer in order to keep the base mediawiki image sufficiently recent). That's not a deal-breaker, and is something we would probably adopt for any of the options discussed here, but worth noting.

In short, this approach offloads special-casing from the service (i.e., init-container clone or hostPath mount dependency) to the image build process.

I don't think either of those is innately better or worse than the other, and the more important question to ask is whether one of them overlaps with other use cases we either already have or want to establish (e.g., like the "init-container approach for self-contained environment setup" being a precursor to what's proposed in T386246#10677776).

In the interest of having a solution ready in the very near term, I'd like to propose that we move ahead with the init-container based approach described in T386246#10572432. The main benefit of this approach, IMO, is that it is highly effective in constraining special-casing to the service itself rather than the environment.

As previously noted, there are two (related) issues that need mitigated:

1. It puts gerrit in the critical path for pod startup, since the the init-container must clone mediawiki/services/parsoid.
2. We have to be careful about how this interacts with scap deployments - e.g., transient gerrit unavailability should not result in a failed scap deployment.

After speaking with @cscott the other day, it sounds like it should be acceptable if parsoidtest (1) can always be updated to the latest mediawiki image on demand (e.g., if a testing-relevant backport is needed) but (2) by default tracks the production mediawiki image asynchronously (e.g., updating at least once every day).

If that's acceptable, then I think that gives us everything we need to move forward with something like the following:

• We'll create a new mw-parsoid-testing namespace (note: placeholder name), containing a single helmfile release that uses the chart changes in https://gerrit.wikimedia.org/r/1121454.
• The sole release in mw-parsoid-testing will be configured with deploy: false in mediawiki-deployments.yaml, so that scap provides a /etc/helmfile-defaults/mediawiki/release values file for its consumption, but (at least for now) does not deploy mw-parsoid-testing.
• We'll create a systemd timer that runs on the active deployment host that ensures mw-parsoid-testing is updated at least once per day, Monday through Thursday.

This both vastly reduces the frequency of pod startups and keeps the service out of the critical path in scap.

With this approach, the "setup" phase of an RT testing run at a new commit involves two steps:

1. Merge a one-line change to helmfile.d/services/mw-parsoid-testing/value.yaml in deployment-charts that points to the new commit-to-test.
2. Run helmfile to apply the change (this also picks up the latest mediawiki image).

The reason for #1 is that we need to persist the commit-to-test somewhere if we're going to support unattended updates to the mediawiki image. Although we could do that in a way where state exists only in k8s objects rather than in deployment-charts (e.g., a configmap external to the release), that's rather complex and has undesirable behavior in the event of a misconfiguration (i.e., updating the commit is decoupled from updating the release itself, so helm cannot meaningfully revert).

The missing piece would be one-off testing via the parse.php maintenance script [0]. For that, there are a couple of different directions we could go.

IMO, it would not be good to fork mwscript-k8s and all supporting infrastructure (e.g., completed job cleanup) to operate in the mw-parsoid-testing namespace.

Instead, I would propose that we keep one-off maintenance scripts in mw-script, and extend mwscript-k8s with a --parsoid-test-commit flag (note: placeholder name) that would add the additional values necessary to enable the parsoid-testing logic in the synthesized values file.

[0] https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing#Running_Parsoid_tools_on_parsoidtest1001

We decided to update parsoidtest1001 to PHP 8.1 "in place" for T380485: Transition parsoidtest1001 to PHP 8.1, so this work no longer transitively blocks 8.1 migration. This will give us more time to develop a solution for this use case on k8s that maximizes overlap / reuse with T276994: Provide an mwdebug functionality on kubernetes (mw-experimental).

@effie and @Jgiannelos have an idea to re-utilize mw-experimental for this. I 've submitted a Cross Team engineering proposal for this.

I have updated the task description, to reflect how we could potentially move this forward, based on discussions during the WE5+WE6 offsite

• ClusterConfig.php has $this->cluster = $_SERVER['SERVERGROUP'] ?? '';
• CommonSettings.php has if ( ClusterConfig::getInstance()->isParsoid() )

We should make the appropriate changes so this will point to the mw-parsoid deployment.

Change #1237472 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mw-parsoid: repurpose for parsoidtest use

https://gerrit.wikimedia.org/r/1237472

Heads up: We should make sure that devs don't rely on home folders as a long term storage for scripts and stuff. Eventually the host node might be recycled.

Heads up: We should orchestrate a helmfile deploy early in our rt-test script to run a deployment to bring the env up to date with the latest.

Change #1238345 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kubernetes::mediawiki_experimental: add parsoid repo

https://gerrit.wikimedia.org/r/1238345

Change #1238349 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] deployment_server: add parsoid pinkllama release

https://gerrit.wikimedia.org/r/1238349

Heads up: We should make sure that we run rt test to the same cluster we send requests. In the new scenario we will have 1 env in eqiad and 1 in codfw.

Change #1238355 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mediawiki: mount parsoid-testing via hostPath

https://gerrit.wikimedia.org/r/1238355

Change #1239169 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] validating-admission-policies: add policy for hostPath mounts for parsoid-testing #

https://gerrit.wikimedia.org/r/1239169

Change #1239170 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] validating-admission-policies: add policy for hostPath mounts for parsoid-testing #0

https://gerrit.wikimedia.org/r/1239170

Change #1239174 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] admin_ng: add hostPath support to mw-parsoid

https://gerrit.wikimedia.org/r/1239174

Thanks, @jijiki!

A couple of thoughts while reviewing yesterday's patch series, that don't really fit on one specific patch:

1. As recently as a couple of weeks ago, mw-parsoid was still receiving very rare requests from restbase. Is that no longer the case? (i.e., that request path has been fully turned down)
2. One of the original requirements (#5) involves the ability to run maintenance scripts under the parsoid-testing mount path. What's our story for that? Some incantation of kubectl exec?
3. Before the last patch in the series (https://gerrit.wikimedia.org/r/1237472) is merged and the new helmfile release is brought up, it seems like it would probably be a safer order of operations to make sure the config overrides in CommonSettings.php are extended to cover it first - i.e., this is how parsoidtest is forced into RO. I might be misunderstanding when you intend to mount in /srv/parsoid-testing, though.
4. If I'm understanding the last patch correctly, it looks like the parsoid testing release would also mount the experimental /srv/mediawiki. Is that correct? If so, my (potentially out-of-date) understanding is that RT testing runs for many hours, during which anyone could unknowingly start using mw-experimental and making local changes, which could disrupt testing.
5. Can we remove the paging configuration and httpbb checks from the service catalog before any work begins on mw-parsoid?

Regarding 1: It shouldn't receive any requests from RB because RB is only running mathoid. Is the traffic so low that could be healthchecks from x-amples ?

Regarding 2: I think we need the ability to run maintenance scripts for debugging purposes.
Regarding 4: We can put some instrumentation in our testing script that checks out a specific version of mediawiki, same way we are doing it for parsoid.

Change #1239651 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] service.yaml: switch mw-parsoid to lvs_setup #2

https://gerrit.wikimedia.org/r/1239651

In T386246#11615689, @Scott_French wrote:

Thanks, @jijiki!

A couple of thoughts while reviewing yesterday's patch series, that don't really fit on one specific patch:

1. As recently as a couple of weeks ago, mw-parsoid was still receiving very rare requests from restbase. Is that no longer the case? (i.e., that request path has been fully turned down)

hieradata/role/common/restbase/production.yaml still has mw-parsoid defined as a listener, those are probably envoy health checks.

Change #1239170 abandoned by Effie Mouzeli:

[operations/deployment-charts@master] validating-admission-policies: add /srv/parsoid-testing #0

Reason:

reworked

https://gerrit.wikimedia.org/r/1239170

Change #1239692 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/mediawiki-config@master] parsoid: Allow overriding special testing config for both host and servergroup

https://gerrit.wikimedia.org/r/1239692

Change #1239695 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] validating-admission-policies: add /srv/parsoid-testing #0

https://gerrit.wikimedia.org/r/1239695

In T386246#11615689, @Scott_French wrote:

Τhank you Scott!

A couple of thoughts while reviewing yesterday's patch series, that don't really fit on one specific patch:

3. Before the last patch in the series (https://gerrit.wikimedia.org/r/1237472) is merged and the new helmfile release is brought up, it seems like it would probably be a safer order of operations to make sure the config overrides in CommonSettings.php are extended to cover it first - i.e., this is how parsoidtest is forced into RO. I might be misunderstanding when you intend to mount in /srv/parsoid-testing, though.

That is right, we submitted 1239692 today, taking care of that

4. If I'm understanding the last patch correctly, it looks like the parsoid testing release would also mount the experimental /srv/mediawiki. Is that correct? If so, my (potentially out-of-date) understanding is that RT testing runs for many hours, during which anyone could unknowingly start using mw-experimental and making local changes, which could disrupt testing.

After discussing with @Jgiannelos we decided against it. The capability is there and can be used on demand, since the ValidatingAdmissionPolicy includes /mnt/mediawiki, and it can be mounted via toggling values.yaml.

5. Can we remove the paging configuration and httpbb checks from the service catalog before any work begins on mw-parsoid?

Yes we can! Submitted under 1239651

Change #1239709 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] restbase::production: remove mw-parsoid listener

https://gerrit.wikimedia.org/r/1239709

In T386246#11620033, @Clement_Goubert wrote:

In T386246#11615689, @Scott_French wrote:

Thanks, @jijiki!

A couple of thoughts while reviewing yesterday's patch series, that don't really fit on one specific patch:

1. As recently as a couple of weeks ago, mw-parsoid was still receiving very rare requests from restbase. Is that no longer the case? (i.e., that request path has been fully turned down)

hieradata/role/common/restbase/production.yaml still has mw-parsoid defined as a listener, those are probably envoy health checks.

Submitted[[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1239709 | 1239709 ]], thanks for finding it !

Change #1239169 merged by jenkins-bot:

[operations/deployment-charts@master] validating-admission-policies: add /srv/parsoid-testing (vanilla)

https://gerrit.wikimedia.org/r/1239169

Change #1239174 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1

https://gerrit.wikimedia.org/r/1239174

Change #1239695 merged by jenkins-bot:

[operations/deployment-charts@master] validating-admission-policies: add /srv/parsoid-testing #0

https://gerrit.wikimedia.org/r/1239695

Thank you both!

In T386246#11619691, @Jgiannelos wrote:

Regarding 2: I think we need the ability to run maintenance scripts for debugging purposes.

Great, thank you for confirming. This will need some thought - i.e., we can presumably make this work via kubectl exec, but we'll need to document (and / or wrap) it well, since the ergonomics change.

One point of note: However we do that, we'll want to make sure that ClusterConfig::getCluster() as used in https://gerrit.wikimedia.org/r/1239692 continues to work as expected. It does right now in mw-script (i.e., returns what would be the SERVERGROUP), so as long as we do something similar, we should be fine (same goes for isParsoid, which we currently need to play some tricks with on parsoidtest1001).

Change #1238345 merged by Effie Mouzeli:

[operations/puppet@production] kubernetes::mediawiki_experimental: add parsoid repo #3

https://gerrit.wikimedia.org/r/1238345

Change #1239651 merged by Effie Mouzeli:

[operations/puppet@production] service.yaml: remove alerts from mw-parsoid #2

https://gerrit.wikimedia.org/r/1239651

Change #1239709 merged by Effie Mouzeli:

[operations/puppet@production] restbase::production: remove mw-parsoid listener

https://gerrit.wikimedia.org/r/1239709

Change #1238349 merged by Effie Mouzeli:

[operations/puppet@production] deployment_server: add parsoid pinkllama release #4

https://gerrit.wikimedia.org/r/1238349

Change #1238355 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: mount parsoid-testing via hostPath #5

https://gerrit.wikimedia.org/r/1238355

Mentioned in SAL (#wikimedia-operations) [2026-02-19T11:29:46Z] <jiji@deploy2002> Started scap sync-world: switching mw-parsoid to pinkllama releases (T386246)

Mentioned in SAL (#wikimedia-operations) [2026-02-19T11:30:49Z] <jiji@deploy2002> jiji: switching mw-parsoid to pinkllama releases (T386246) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-02-19T11:34:21Z] <jiji@deploy2002> Finished scap sync-world: switching mw-parsoid to pinkllama releases (T386246) (duration: 06m 12s)

Change #1237472 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: repurpose for parsoidtest use #6

https://gerrit.wikimedia.org/r/1237472

Change #1240663 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mw-parsoid: repurpose for parsoidtest use #7

https://gerrit.wikimedia.org/r/1240663

Change #1240663 abandoned by Effie Mouzeli:

[operations/deployment-charts@master] mw-parsoid: repurpose for parsoidtest use #7

Reason:

merge conflict

https://gerrit.wikimedia.org/r/1240663

Change #1240665 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/deployment-charts@master] mw-parsoid: repurpose for parsoidtest use #7

https://gerrit.wikimedia.org/r/1240665

Change #1240665 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: repurpose for parsoidtest use #7

https://gerrit.wikimedia.org/r/1240665

Change #1239692 merged by jenkins-bot:

[operations/mediawiki-config@master] parsoid: Override test config for parsoid testing env on k8s

https://gerrit.wikimedia.org/r/1239692

Mentioned in SAL (#wikimedia-operations) [2026-02-19T14:18:44Z] <jgiannelos@deploy2002> Started scap sync-world: Backport for [[gerrit:1239692|parsoid: Override test config for parsoid testing env on k8s (T386246)]]

Mentioned in SAL (#wikimedia-operations) [2026-02-19T14:20:54Z] <jgiannelos@deploy2002> jgiannelos: Backport for [[gerrit:1239692|parsoid: Override test config for parsoid testing env on k8s (T386246)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-02-19T14:26:07Z] <jgiannelos@deploy2002> Finished scap sync-world: Backport for [[gerrit:1239692|parsoid: Override test config for parsoid testing env on k8s (T386246)]] (duration: 07m 23s)

Change #1240750 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] x-wikimedia-debug-routing: add routing to mw-parsoid

https://gerrit.wikimedia.org/r/1240750

Change #1240750 merged by Effie Mouzeli:

[operations/puppet@production] x-wikimedia-debug-routing: add routing to mw-parsoid

https://gerrit.wikimedia.org/r/1240750

@Jgiannelos and I ran some tests and did some troubleshooting today, which eventually got mw-parsoid to a good state, which is also now accessible via XWD

date: Fri, 20 Feb 2026 12:10:12 GMT
server: mw-parsoid.eqiad.pinkllama-c4959747f-d9ncl

We will complete a round of testing. The remaining work includes documentation, dashboards, team usability testing, and tying up any remaining loose ends.

Change #1251192 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-parsoid: Delete values-canary.yaml

https://gerrit.wikimedia.org/r/1251192

Change #1251192 merged by jenkins-bot:

[operations/deployment-charts@master] mw-parsoid: Delete values-canary.yaml and values-migration.yaml

https://gerrit.wikimedia.org/r/1251192

Can I confirm what is remaining to close this?

In T386246#11717877, @MLechvien-WMF wrote:

Can I confirm what is remaining to close this?

We were missing the documentation and monitoring