In T411058#11565014, @KartikMistry wrote:

I'm still debugging, and probably best way to check with reverting original memory allocation. Patch is coming up.

Waiting to get some updates from T411058 after staging deployment.

I'm still debugging, and probably best way to check with reverting original memory allocation. Patch is coming up.

I'm also investigating the MinT instance's frequent downtime, so recreating the instance creation is taking longer than usual. I'll update my findings here.

In T402882#11414845, @abi_ wrote:

$ ssh lpl-mleb-master2.language.eqiad1.wikimedia.cloud
ssh: Could not resolve hostname lpl-mleb-master2.language.eqiad1.wikimedia.cloud: Name or service not known

Any ideas what could be wrong?

Thanks @jijiki

@RLazarus We deployed MinT lastly on 06 Nov with a37ece7cde26383bba8b3f22519635f3e3b95da5. Is it possible that resource allocation is mismatch after that?

MediaWiki Job runner is enabled in both MLEB instances.

Possible to do the removal of the key this week? @jijiki

In T402882#11357281, @KartikMistry wrote:

Both MLEB instances are unstable with low RAM. I'll be recreating them.

From Apertium service log: https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.11.19?id=SQ7NmZoBlLUf3R62SA4t doesn't say much more than Translation error in pair rus-bel: Stream is closed and then A pipe for pair rus-bel seems stuck, scheduling restart at https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-k8s-1-7.0.0-1-2025.11.19?id=VA7NmZoBlLUf3R62SQ9G

@jijiki, Possible to recheck the current status? I've upgraded Apertium today.

Can we remove the key? Possibly unused key is causing: Could not parse cxserver.conf.mt.Yandex errors.

Both MLEB instances are unstable with low RAM. I'll be recreating them.

Note that new instances are lpl-mleb-master2.language.eqiad1.wikimedia.cloud and lpl-mleb-stable2.language.eqiad1.wikimedia.cloud

We can also take a relook at OOM issue here when T408515 is done.

In T402882#11311370, @abi_ wrote:

We will need to install the Elasticsearch plugin: https://github.com/wikimedia/search-extra

In T402882#11311364, @abi_ wrote:

Both MLEB instances currently fatal out when loading Special:Version:

[b603bde6e0f54d5370a502d9] /index.php/Special:Version Error: Class "Elastica\Client" not found

Backtrace:

from /var/www/core/extensions/Elastica/includes/ElasticaConnection.php(97)
#0 /var/www/core/extensions/Elastica/includes/ElasticaConnection.php(68): MediaWiki\Extension\Elastica\ElasticaConnection->getClient()
#1 /var/www/core/extensions/CirrusSearch/includes/Connection.php(130): MediaWiki\Extension\Elastica\ElasticaConnection->setConnectTimeout()
#2 /var/www/core/extensions/CirrusSearch/includes/Hooks.php(535): CirrusSearch\Connection->__construct()

See: https://lpl-mleb-stable.wmcloud.org/index.php/Special:Version

The error 410 shows that the API is permanently gone. From the documentation of the API, we were using (version 1.5) is no longer mentioned, and the 2.0 api and API URL path is also different.

There are some issues with DNS making MLEB accessible using subdomains like stable.lpl-mleb.wmcloud.org. I'll take approach that's better for the long term.

It seems that Yandex is no longer accessible. See: T407345

In T394982#11236915, @Nikerabbit wrote:

I cannot reproduce locally using master branch of cxserver.

In the logs I would expect to see something like below. But because we don't, it fails while outputting this line or before it:

{"@timestamp":"2025-10-02T12:56:26.511Z","ecs.version":"8.10.0","http":{"request":{"id":"ea22f5a8-2001-4f62-a372-73dd95389edd","method":"GET"}},"log.level":"debug","message":"Getting page en:Albert Einstein for fr","service":"cxserver-dev","url":{"path":"/v2/page/en/fr/Albert%20Einstein"}}

Holding on updating lpl-services till we are done with T394982

In T405191#11227946, @abi_ wrote:

Looks good.

A few other notes:

1. Two dashboards for Recommendation API that can be merged into one.
2. Two dashboards for MinT that can be merged into one.
3. Normalize the title for various dashboard. Some use small letters with hyphen (cxserver-last-24-hours), and some are using capitals and spaces: MinT ECS Last 24 Hours

Staging logs look like this:

Updated dashboard description so that searching by LPL keyword will list all dashboards now.

During testing, I found that the page API is broken.