Background
To support Pretrain, we'll need to divert MediaWiki API traffic at REST Gateway - i.e., testwiki-bound traffic that would normally flow to the upstream mw-api-ext-ro cluster will instead need to flow to mw-pretrain.
Historically, to implement this kind of diversion, we've used dynamic metadata extraction and static LB endpoint subsetting - see, e.g., what we did to support PHP_ENGINE cookie diversion (see https://gerrit.wikimedia.org/r/1194790) or, much earlier, X-Wikimedia-Debug diversion (see https://gerrit.wikimedia.org/r/627588; note that this is not currently used / supported in REST Gateway).
As can be seen in the former, there's a clear maintainability problem with this approach: appropriately handling the explosion of possible metadata value states in the static LB endpoint filter metadata - particularly if we want to simultaneously support diversion for Pretrain, X-Wikimedia-Debug (see T428909#12070270), PHP_ENGINE, and (maybe) some form of transitive RO/RW routing (see T401396#11383272).
Further, note that there's an observability shortcoming: Using LB endpoint subsetting obscures the true upstream, since all potential selections are modeled as endpoints of one "logical" cluster (e.g., everything would still appear to be mw-api-ext-ro).
Is there a better way?
I believe the answer is yes, in the form of the cluster specifier Lua plugin, which would give us a clearer, more easily testable strategy for route-level upstream cluster selection than we could ever achieve with LB endpoint subsetting. Notably, there is plenty of prior art that integrates Lua plugins for various aspects of request processing in REST Gateway - i.e., this is not without precedent, and there are patterns upon which we can build.
If we go this route, then this is also an ideal time to clean up / remove the existing metadata-driven logic as well, and we should aim to coordinate with work in T428625.