"Too many open files" should be a transient problem. Searching for "Too many open files" in the last 7 days of beta cluster logging (https://beta-logs.wmcloud.org/goto/1a495e8c49d87cef66f0bdecdcd739bf) shows that it happens on deployment-mediawiki14 (the only web request facing MediaWiki in beta right now) in bursts. This could just be a sign of traffic patterns that overload the instance.

It is also reasonable to consider that there is some ulimit tweak that happens for the php-fpm process in production that isn't applied on the deployment-prep instances, but a quick git grep -l LimitNOFILE in ops/puppet.git doesn't turn up any obvious candidates.

Rechecking that happens once per hour.

Prod baremetal had the limit overriden in the systemd unit:

[Service]
...
<%- if @increase_open_files -%>
LimitNOFILE=1024000
LimitNOFILESoft=1024000
<%- end -%>

Which was made True for jobrunners with 764a3d1d23964517eb761d9f2e6d6ccf9efe1f8a / T344428.

For MediaWiki on Kubernetes I have found in operations/deployment-charts:

LimitNOFILE=1048576
LimitNOFILESoft=1048576

The php8.1-fpm.service unit has:

LimitNOFILE=524288
LimitNOFILESoft=1024

That is the systemd default according to /etc/systemd/system.conf: #DefaultLimitNOFILE=1024:524288.

The still existing php7.4-fpm.service one had:

LimitNOFILE=1048576
LimitNOFILESoft=1048576

I don't know why or how it was raised, maybe the system.conf was set differently for MediaWiki app servers.

Anyway the fix is to apply to the deployment-mediawiki Puppet prefix:

profile::mediawiki::php::increase_open_files: True

Mentioned in SAL (#wikimedia-releng) [2025-04-10T14:16:22Z] <hashar> deployment-prep: profile::mediawiki::php::increase_open_files: True on https://horizon.wikimedia.org/project/prefixpuppet/?tab=prefix_puppet__puppet-deployment-mediawiki # T389422

I ran Puppet on deployment-mediawiki14, the systemd override has been reloaded and:

$ systemctl show php8.1-fpm.service|grep LimitNO
LimitNOFILE=1024000
LimitNOFILESoft=1024000

The LimitNOFILE was thus raised from 524288 to 1024000 which should solve the issue.

Looking at the unit it complains about a non existing LimitNOFILESSoft:

Apr 10 14:17:36 deployment-mediawiki14 systemd[1]: /etc/systemd/system/php8.1-fpm.service.d/puppet-override.conf:7: Unknown key name 'LimitNOFILESoft' in section 'Service', ignoring.

LimitNOFILESoft is in the DBUS specification but systemd uses <soft>:<hard> to define the limits (see https://github.com/systemd/systemd/pull/1994 ) and from systemd.exec(5):

Resource limits may be specified in two formats:

• either as single value to set a specific soft and hard limit to the same value,

or

• as colon-separated pair soft:hard to set both limits individually (e.g. "LimitAS=4G:16G").

Use the string infinity to configure no limit on a specific resource.
The multiplicative suffixes K, M, G, T, P and E (to the base 1024) may be used for resource limits measured in bytes (e.g. "LimitAS=16G").

Change #1135743 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] mediawiki: remove unsupported LimitNOFILESoft directive

https://gerrit.wikimedia.org/r/1135743

Change #1135743 merged by Clément Goubert:

[operations/puppet@production] mediawiki: remove unsupported LimitNOFILESoft directive

https://gerrit.wikimedia.org/r/1135743

This can be close in a few days after we have it is confirmed the Too many open files errors are gone: https://beta-logs.wmcloud.org/goto/1a495e8c49d87cef66f0bdecdcd739bf

Thank you for the verification and I am quite happy we have managed to fix that one!