I just deployed this change (with config), and it looks good as per the test plan.

I'm able to start the article en:Reptile as used as an example by @Chronus

Testing done with cloud instance: https://cxserver.wmcloud.org/v2?doc#/Page%20content/get_v1_page__language___title___revision_

In T405004#11859378, @KartikMistry wrote:

We will also observe this during nodejs24 migration of cxserver, but it shouldn't affect the overall.

In T404298#11919315, @Chronus wrote:

For translations from English to Portuguese (whether already in progress or new), the following error appears: “Critical error: Failed to load the content translation due to an internal error.”

Thanks a lot @Marostegui and @Dzahn

If it can be a good place, https://office.wikimedia.org/wiki/User:KMistry_(WMF)#New_key I've put it here as well.

In T425853#11908952, @Dzahn wrote:

Could you maybe drop a file in some home directory on a production server that confirms it?

Any place, just let us know where.

We will also observe this during nodejs24 migration of cxserver, but it shouldn't affect the overall.

In T405004#11792546, @KartikMistry wrote:

I'm debugging a few items for this:

• Is the MW API overloaded while doing requests?
• Is there any proxy (6500) instability?
• Connection pool exhaustion
• Timeout mismatch between cxserver and rec-api?
• No retry?

@Clement_Goubert Hi, do we need to change anything on the recommendation-api side for this change?

I'm debugging a few items for this:

Although the Google Translation API does not seem to support Central Kanuri yet. So, closing this task.

It seems Google is enabled as 'kr' and 'knc' is mapped with 'kr' in the lib/mt/Google.js but Google doesn't appear as a supported MT in the Content Translation tool.

Pods look fine so far:

In T411058#11721786, @JMeybohm wrote:

In T411058#11721747, @KartikMistry wrote:

@RLazarus After reducing replicas, I was able to deploy MinT in codfw. How to delete failing older pods from the codfw now?

I've cleaned them up in codfw. Please keep in mind to align deployments in codfw and eqiad (right now the replicas in both clusters differ).

@RLazarus After reducing replicas, I was able to deploy MinT in codfw. How to delete failing older pods from the codfw now?

@santhosh After deploying https://cxserver.wikimedia.org/v2/page/en/tr/Koch_Media seems still broken, while https://cxserver.wikimedia.org/v2/page/fi/smn/Alexander_Stubb fetches page successfully.

https://translate.wmcloud.org/ is now running Trixie Debian version.

Testing some performance tweaks from https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1248388/3/helmfile.d/services/machinetranslation/values.yaml to the cloud instance.

In T408160#11676356, @MusikAnimal wrote:

Looks like some capacity problems are still being worked on based on T411058: Can't deploy machinetranslation due to exceeding resource quotas. I guess we should wait for that to be resolved, then we can re-ping LPL to see if it's a good time to switch on machine translation by default.

Knowing there are capacity issues, I want to first improve performance on our end.

Noting that I think we've now got our implementation as efficient as it can be (r1204089). Any given wish typically makes up to 4 translation requests at most.

This is blocking passing of CI for recommendation-api as well, where we rely on waiting for 30 minutes: https://integration.wikimedia.org/ci/job/recommendation-api-ng-pipeline-test/773/console

In T411058#11565014, @KartikMistry wrote:

I'm still debugging, and probably best way to check with reverting original memory allocation. Patch is coming up.

Waiting to get some updates from T411058 after staging deployment.

I'm still debugging, and probably best way to check with reverting original memory allocation. Patch is coming up.

I'm also investigating the MinT instance's frequent downtime, so recreating the instance creation is taking longer than usual. I'll update my findings here.

In T402882#11414845, @abi_ wrote:

$ ssh lpl-mleb-master2.language.eqiad1.wikimedia.cloud
ssh: Could not resolve hostname lpl-mleb-master2.language.eqiad1.wikimedia.cloud: Name or service not known

Any ideas what could be wrong?

Thanks @jijiki

@RLazarus We deployed MinT lastly on 06 Nov with a37ece7cde26383bba8b3f22519635f3e3b95da5. Is it possible that resource allocation is mismatch after that?

MediaWiki Job runner is enabled in both MLEB instances.

Possible to do the removal of the key this week? @jijiki

In T402882#11357281, @KartikMistry wrote:

Both MLEB instances are unstable with low RAM. I'll be recreating them.

From Apertium service log: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.11.19?id=SQ7NmZoBlLUf3R62SA4t doesn't say much more than Translation error in pair rus-bel: Stream is closed and then A pipe for pair rus-bel seems stuck, scheduling restart at https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.11.19?id=VA7NmZoBlLUf3R62SQ9G