Was this cleared with the SREs? According to https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters#staging:

Other uses of this cluster [staging] than the one described above (e.g. as a development environment, a CI runner, a quality assurance platform or a demo scene to name a few) MUST NOT be allowed.

The proposed usecase seems to fall under a "quality assurance platform" and/or "a demo scene" and as such, it appears to be explicitly prohibited by the policy above.

In T407029#11372792, @Urbanecm_WMF wrote:

Was this cleared with the SREs? According to https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters#staging:

Other uses of this cluster [staging] than the one described above (e.g. as a development environment, a CI runner, a quality assurance platform or a demo scene to name a few) MUST NOT be allowed.

The proposed usecase seems to fall under a "quality assurance platform" and/or "a demo scene" and as such, it appears to be explicitly prohibited by the policy above.

It was. It is broadly acknowledged that this is very much not ideal. And while the proposed usage does have QA-aspects to it, it also has staging aspects. After verifying that things work with the staging cluster as expected, the plan is to move forward to using the production cluster with testwiki. But this is certainly still not the same thing as "proper" staging usage.

The lack of a dedicated test/demo/QA/(dev) environment is probably one of the main learnings from this collaboration. For now, we must make do with what we have.

Since I find myself constantly looking for it:

xref T401021: Data Persistence Design Review: Improve Tone Suggested Edits newcomer task

xref T407356: Create Production Revise Tone Recommendation Provider

Example staging request:

[urbanecm@deploy2002 ~]$ curl -s https://data-gateway.k8s-staging.discovery.wmnet:30443/public/ml_cache/page_paragraph_tone_scores/testwiki/168753/680406 | json_pp
{
   "rows" : [
      {
         "content" : "Schwarzwald Castle offers a wonderful glimpse into medieval life with well-preserved ruins and informative guided tours available in multiple languages. The scenic hilltop location provides excellent views of the surrounding Black Forest, making it a popular choice for families and history enthusiasts. The on-site museum showcases an impressive collection of medieval artifacts, and the castle grounds are perfect for a relaxing afternoon visit. With convenient parking and a charming café serving local specialties, Schwarzwald Castle makes for an enjoyable day trip from Freiburg.",
         "idx" : -1,
         "model_version" : "v1",
         "page_id" : 168753,
         "revision_id" : 680406,
         "score" : 0.886,
         "wiki_id" : "testwiki"
      }
   ]
}
[urbanecm@deploy2002 ~]$

@Michael and I discussed this task today (before it was re-purposed to focus on the production Data Gateway). I discovered that the MediaWiki containers aren't allowed to talk to the staging endpoint (eg. data-gateway.k8s-staging.discovery.wmnet), and requests to that domain timeout:

[urbanecm@deploy2002 ~]$ mwscript-k8s --attach -- shell.php --wiki=testwiki
ℹ️ Your job will run on PHP 8.3 (T405955). If you encounter a compatibility issue, you can use --php_version 8.1 to explicitly select 8.1.
⏳ Starting shell.php on Kubernetes as job mw-script.codfw.twtofb82 ...
🚀 Job is running.
ℹ️ Expecting a prompt but don't see it? Due to a race condition, the beginning of the output might be missing. Try pressing enter.
📜 Attached to stdin/stdout:


> $hrf = \MediaWiki\MediaWikiServices::getInstance()->getHttpRequestFactory()
= MediaWiki\Http\HttpRequestFactory {#5490}

> $r = \GrowthExperiments\Util::getJsonUrl($hrf, 'https://data-gateway.k8s-staging.discovery.wmnet:30443/public/ml_cache/page_paragraph_tone_scores/testwiki/168753/680406')
= MediaWiki\Status\Status {#5477
    +value: 0,
    +success: [],
    +successCount: 0,
    +failCount: 0,
    +statusData: null,
    +cleanCallback: false,
  }

> $r->getErrors()
= [
    [
      "type" => "error",
      "message" => "http-timed-out",
      "params" => [
        "https://data-gateway.k8s-staging.discovery.wmnet:30443/public/ml_cache/page_paragraph_tone_scores/testwiki/168753/680406",
      ],
    ],
    [
      "type" => "error",
      "message" => "http-bad-status",
      "params" => [
        "0",
        "Error",
      ],
    ],
  ]

> ^D

   INFO  Ctrl+D.

[urbanecm@deploy2002 ~]$

In addition to that, direct conversation between services seems to be discouraged (if not prohibited). Normally, such communication happens via Service Proxies. Adding a listener involves adding the service [to the list of listeners)(https://gerrit.wikimedia.org/g/operations/puppet/%2B/refs/heads/production/hieradata/common/profile/services_proxy/envoy.yaml). This also adjust the network policies (to allow the communication to happens). However, this list doesn't make it possible to declare a listener to a service running in another cluster.

Overall, using the staging instance appears to be significantly more cumbersome from an infrastructure perspective.

In T407029#11380800, @Michael wrote:

In T407029#11372792, @Urbanecm_WMF wrote:

Was this cleared with the SREs? [...]

It was. It is broadly acknowledged that this is very much not ideal. [...]

To clarify and for posterity: Based on off-Phab interactions with @Michael, this was discussed with Data Persistence SREs (via Slack DMs), but not with Service Operations (who are the cluster owners here).

Change #1207886 had a related patch set uploaded (by Michael Große; author: Michael Große):

[operations/mediawiki-config@master] testwiki: enable ReviseTone experiment

https://gerrit.wikimedia.org/r/1207886

Change #1207886 merged by jenkins-bot:

[operations/mediawiki-config@master] testwiki: enable ReviseTone experiment

https://gerrit.wikimedia.org/r/1207886

Mentioned in SAL (#wikimedia-operations) [2025-11-24T15:34:44Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1207886|testwiki: enable ReviseTone experiment (T407029)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-24T15:39:24Z] <urbanecm@deploy2002> urbanecm, migr: Backport for [[gerrit:1207886|testwiki: enable ReviseTone experiment (T407029)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-24T15:47:03Z] <urbanecm@deploy2002> Finished scap sync-world: Backport for [[gerrit:1207886|testwiki: enable ReviseTone experiment (T407029)]] (duration: 12m 19s)