Background
Our plan of record is to use k8s Ingress to route testwiki traffic to the associated Pretrain deployments - i.e., direct traffic bound for test.wikipedia.org to the appropriate services by path-dependent use case (generic web or API, jobrunner, etc.).
While Ingress is ideal for this kind of application, and converges toward ideas explored to enable Single Version MediaWiki, we're not yet at a point where we want put Ingress in the critical path for the existing set of MW-on-k8s services (e.g., mw-web).
What that means is that we must externally divert testwiki-bound traffic to Ingress.
Traffic diversion paths
To support this, we need to support traffic diversion on a number of different request paths:
- Divert external testwiki traffic to Ingress at ATS.
- Note that this is limited to cases where ATS is the last-hop toward MW-on-k8s.
- In practical terms, that means generic web traffic and X-Wikimedia-Debug traffic (we will have a Pretain-debug), and not traffic diverted to REST Gateway.
- Divert external testwiki traffic to Ingress at REST Gateway.
- In practical terms, this means writing envoy configuration to divert to a different upstream cluster based on Host. This is the focus of T433752, together with supporting other forms of traffic diversion (e.g., based on the PHP_ENGINE cookie).
- Divert internal testwiki traffic to Ingress.
- In practical terms, this is mainly focused on internal API traffic and jobrunner traffic, which will again rely on Host-header diversion via the envoy service mesh (i.e., for all services where the relevant listener(s) are enabled). We expect this to cover the vast majority of internal traffic.
- One area where that gets tricky is changeprop. Although the jobqueue deployment thereof now also applies the appropriate Host header to outbound requests (T395451), we don't actually use the service mesh on either. Unless we were to implement diversion directly in changeprop (strongly not prefered), the most straightforward approach is to imbue them with service mesh. This, for both changeprop (as a client of the MediaWiki API) and changeprop-jobqueue (as a client of the jobrunners), is the focus of T434925.
- Another area that may need further investigation is internal API traffic from Knative workloads running in the ml-serve k8s clusters, which appear to use a completely different mechanism to proxy egress HTTP to internal services (see e.g. net_istio.mesh.service_entries). It has yet to be seen whether this must be handled in some form (e.g., whether these workloads even produce testwiki-bound requests).
- In practical terms, this is mainly focused on internal API traffic and jobrunner traffic, which will again rely on Host-header diversion via the envoy service mesh (i.e., for all services where the relevant listener(s) are enabled). We expect this to cover the vast majority of internal traffic.
Nearly all of these will require deployment in two phases: one phase introducing new features to support traffic diversion and then a second phase where we enable said diversion for testwiki. Once that work is complete, the logging functionality that will be developed in T439144 can be enabled, and we can begin learning what we may have missed.
Work tracked directly in this task primarily covers horizontal routing concerns such diversion in the service mesh, as well as executing the traffic migration using the implementations developed here and in subtasks.
Current status of worked tracked here
Current as of 2026-09-30:
- Divert external testwiki traffic to Ingress at ATS.
- x-wikimedia-debug.lua routing supports diversion to mw-pretrain for testwiki traffic, when users select the k8s-mw-pretrain backends (see https://gerrit.wikimedia.org/r/1327621).
- Add these backends to debug.json, so that they're available in the WikimediaDebug extension: https://gerrit.wikimedia.org/r/1346796.
- Consider "aliasing" the more common k8s-mwdebug backends in x-wikimedia-debug.lua to k8s-mw-pretrain equivalents for compatibility when traffic is for testwiki. This will require two changes: (1) x-wikimedia-debug.lua will need updated to support this (for a demo, see: https://gerrit.wikimedia.org/r/1346799) and (2) mw-pretrain's ingress routing will need to become more relaxed in terms of the X-Wikimedia-Debug backends it supports. As I write this, I am not convinced that we actually want this. If we do go this route, we should only do so as traffic is migrated.
- MIGRATE: When we're ready to start migrating traffic, we need to configure ATS to divert test.wikipedia.org traffic to mw-pretrain.
- Merge https://gerrit.wikimedia.org/r/1326930 and deploy like any other ATS config change (coordinate with Traffic).
- x-wikimedia-debug.lua routing supports diversion to mw-pretrain for testwiki traffic, when users select the k8s-mw-pretrain backends (see https://gerrit.wikimedia.org/r/1327621).
- Divert external testwiki traffic to Ingress at REST Gateway.
- Complete the implementation work in T433752.
- MIGRATE: Merge https://gerrit.wikimedia.org/r/1352503 and deploy rest-gateway.
- Divert internal testwiki traffic to Ingress.
- Host-based traffic splitting is now supported in Puppet (i.e., in our envoy listener definitions, as well as bare-metal envoy service mesh use cases; see, e.g., https://gerrit.wikimedia.org/r/1328247, https://gerrit.wikimedia.org/r/1328256), and the relevant deployment-charts mesh modules have been extended to match (see, e.g., https://gerrit.wikimedia.org/r/1331858).
- We have prepared a global upgrade to all mesh-dependent charts to these new module versions, which have been split into three patches for clearer grouping (i.e., of upgraded modules). Once merged, these patches can be Charlie'd out to all affected services.
- Merge https://gerrit.wikimedia.org/r/1338000 to upgrade the mediawiki chart.
- Merge https://gerrit.wikimedia.org/r/1351523 to upgrade the changeprop chart, which unblocks work on T434925.
- Merge https://gerrit.wikimedia.org/r/1338030 to upgrade all other dependent charts, and Charlie it out.
- NOTE: See discussion in the commit message. There's at least one oddity among the module updates pulled in (affecting the turnilo chart), which has been addressed, but is good to be aware of.
- Once this module upgrade is complete, proceed with the work in T434925 for changeprop, and come back here when that's complete.
- MIGRATE: When we're ready to begin internal (mesh) traffic migration:
- For internal API traffic, merge https://gerrit.wikimedia.org/r/1328249. This will (1) enable diversion for all bare-metal use cases and (2) introduce a change to the services proxy data present in /etc/helmfile-defaults on the deployment host. Deploying the latter, thus enabling the functionality previously introduced via the new mesh modules, will require a round of Charlie'ing.
- For jobrunner traffic, merge https://gerrit.wikimedia.org/r/1346775 and deploy changeprop-jobqueue (after running puppet on the deployment host).
Additional discussion
Why not Ingress-all-the-things? - When developing the proposal for the Pretrain MVP environment, we prioritized avoiding significant intrusive architectural changes for the existing MW-on-k8s services. However, if we were to migrate both mw-jobrunner and mw-api-int to Ingress, that would supersede all of the work in #3 (and in many ways result in a simpler-to-reason-about end state - diversion is consolidated in one place rather than being distributed in mesh configuration). One might further argue that if Single-Version MediaWiki is likely to become the future and follow the path set out last year (Ingress as PoR), then maybe this would be valuable regardless. There are many more pros and cons and practicalities than I'm detailing here, but I remain open to a change in approach here.