We noticed that we had a respectable number of errors on codfw the pas few weeks, for example
{F57489243}
Digging deeper we made the following observations, keeping in mind that **eqiad's memcached traffic is ~x3 codfw's**
**restarts**
*// eqiad:// we had almost no restarts
*// codfw: // quite a few pods with 4-5+ restarts recorded
**Allocated fibers**
* //eqiad:// some pods where going over the 10k limit, but nothing worrisome
* //codfw:// many pods where going over the 10k limit, but the pattern also was verifying that we indeed had restarts
[[ https://grafana-rw.wikimedia.org/d/ltSHWhHIk/mw-mcrouter?orgId=1&from=1725408000000&to=1725494399000&var-site=codfw&var-prometheus=k8s&var-memcached_server=All&var-kubernetes_namespace=mw-mcrouter&var-kubernetes_pod_name=All&var-container=mcrouter-main-mcrouter&viewPanel=11 | allocated fibers eqiad ]]
[[ https://grafana-rw.wikimedia.org/d/ltSHWhHIk/mw-mcrouter?orgId=1&from=1725408000000&to=1725494399000&var-site=codfw&var-prometheus=k8s&var-memcached_server=All&var-kubernetes_namespace=mw-mcrouter&var-kubernetes_pod_name=All&var-container=mcrouter-main-mcrouter&viewPanel=11 | allocated fibers codfw ]]
{F57489290}
{F57489292}
Looking at kubernete's nodes logs, we found that mcrouter was getting oom-killed:
`[Wed Sep 4 13:53:44 2024] Memory cgroup out of memory: Killed process 802345 (mcrouter) total-vm:1347040kB, anon-rss:1039776kB, file-rss:18384kB, shmem-rss:0kB, UID:778 pgtables:2364kB oom_score_adj:998`
This should be considered normal, the fact remains that, eqiad serves 3 times this traffic, and does not get have any mcrouter pods oomkilled.
**Update Sept 5 2024: **we doubled the memory limits of mw-mcrouter. The restarts and oomkills have stopped, however, we are still observing errors.
Many thanks to @TK-999 for joining the debugging session for this
description TBA