Page MenuHomePhabricator

Puppet failing on deployment-kafka-jumbo-3 due to memory pressure
Open, Needs TriagePublic

Description

Error: /Stage[main]/Ldap::Client::Nosssd/Package[libnss-sss]: Could not evaluate: Cannot allocate memory - fork(2)
Error: /Stage[main]/Ldap::Client::Nosssd/Package[libsss-sudo]: Could not evaluate: Cannot allocate memory - fork(2)
Error: /Stage[main]/Ldap::Client::Nosssd/Package[sssd]: Could not evaluate: Cannot allocate memory - fork(2)
Error: /Stage[main]/Profile::Wmcs::Instance/Exec[enable_sites_local]: Could not evaluate: Cannot allocate memory - fork(2)
...
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.7G        136M         22M        146M        124M
Swap:            0B          0B          0B

I think this is really just kafka queues holding more messages than the available ram on the instance can support.

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2021-07-09T00:18:38Z] <bd808> Silenced puppet failure alert for deployment-kafka-jumbo-3 for the next 7 days (T286358)