Here's a summary of my findings so far, although keep in mind that I'm well out of my depth here and this is an amalgamation of guesswork.

mw1357 is a server that was depooled and left in the leaky state without being rebooted, to preserve for investigation.

The only thing that shows up in /proc/meminfo as being obviously absurd are a very high number of 4k-sized pages allocated in the TLB. There is an abnormal amount of kernel slab allocator memory shown as being consumed (10x other servers), but it's not nearly enough to account for the usage (~2.4GB).

So since it isn't userspace processes, nor the slab allocator, nor anything else shown in meminfo, that basically has to mean that the issue is lower-level than all that. Beneath the slab allocator, and beneath all physical page allocation in Linux, is the buddy allocator. It reports stats in /proc/buddyinfo and /proc/pagetypeinfo. Each row is a kind of flavor of memory (e.g. NUMA node x, relocate-able); each column within a row is the number of free blocks for a given power-of-two number of pages, starting with 2^0==1 page==4 kB, progressing upwards to 2^10==1024 pages==4MByte of RAM.

On a healthy system without fragmentation you'll see something like this:

Node    0, zone   Normal, type      Movable      0      0      0    407    646    189    201    289    290    179   5272

Lots of large, contiguous blocks of pages are free (> 20 GB worth). There aren't many small blocks of free memory, but that's fine, as splitting larger blocks in half is trivial and fast.

On mw1357, it looks like this:

Node    0, zone   Normal, type      Movable 484607 101153   3866      0      0      0      0      0      0      0      0

This is severe internal fragmentation: the available memory exists only in small 4kB/8kB/16kB blocks, and there's no free memory in any of the larger orders of pages.

I tried sudo sysctl -w vm/compact_memory=1 on mw1357, which forces an attempt at coalescing pages, and while that created more free blocks of small orders, it didn't coalesce any small chunks into larger orders.

At this point I was suspicious of many small objects allocated in the kernel causing this excessive fragmentation.

I looked at slabtop output on mw1357 and saw this:

For comparison, let's also look at slabtop on mw1359, another apiserver that was rebooted and has been serving requests for 9.5 hours: P12234

Things that stand out:

• There's 1M more objects on mw1357 than mw1359, and there's almost 3x the # of slabs on mw1357
• There's many, many more kmalloc-node and kmalloc-32 slabs in use on mw1357, by over an order of magnitude. kmalloc-32 indicates literally calls to kmalloc(32) from within the kernel. There's also a huge increase in kmalloc-4096 nodes, by many orders of magnitude, but I somehow missed this at first.

I did some work to backport bpfcc-tools from Buster to Stretch, so that I could use the memleak tracing tool it provides, which instruments kernel tracepoints and reports stacktraces of still-existent allocations that are older than a specified duration. (More details here and code here.) I installed those packages on mw1359, which is serving traffic, and for about five hours now I've been running memleak-bpfcc -o 60000 -z 31 -Z 33 30, which hooks into kernel tracepoints for kmalloc, and if the allocation was 31<x<33 bytes, saves a stacktrace, and then every 30 seconds, prints out stacktraces of any allocation that has not yet been kfree()d and also is older than 60,000 milliseconds.

Here's the output accumulated over ~5h, started at 16:24: P12237

There's many sets of traces involving memcg_create_kmem_cache and I'm a bit suspicious of it. The set of traces that doesn't involve that, involves kernfs_iop_mkdir.

Anyway, after noticing they are at least as much of a problem as the 32-byte allocations, now I'm going to similarly instrument 4kB allocations and leave that running for some hours.

There's some preliminary results from tracing 4096-byte allocations at

A thing that someone daring in EUTZ might want to try: Using perf probe, or by modifying the bpfcc-memleak script, or by writing a trivial bpftrace script: attach a tracepoint to memcg_schedule_kmem_cache_create and gather calling stacktraces. That's the function that creates the work item that results in a worker thread calling memcg_create_kmem_cache, as seen in the stack traces we saw for 32-byte mallocs.

Mentioned in SAL (#wikimedia-operations) [2020-08-13T08:43:31Z] <_joe_> downgrading imagemagick on mw1378 T260281

In T260281#6381768, @CDanis wrote:

attach a tracepoint to memcg_schedule_kmem_cache_create and gather calling stacktraces. That's the function that creates the work item that results in a worker thread calling memcg_create_kmem_cache, as seen in the stack traces we saw for 32-byte mallocs.

I've installed systemtap on mw1357 and started instrumenting memcg_create_kmem_cache. The function does not seem to be called at present? Same goes for memcg_create_kmem_cache.

root@mw1357:~# stap -ve 'probe kernel.function("memcg_schedule_kmem_cache_create") { print_backtrace() ; exit() }'
Pass 1: parsed user script and 477 library scripts using 153836virt/86440res/6576shr/80088data kb, in 210usr/20sys/241real ms.
Pass 2: analyzed script: 1 probe, 2 functions, 0 embeds, 0 globals using 195668virt/129392res/7460shr/121920data kb, in 560usr/70sys/620real ms.
Pass 3: using cached /root/.systemtap/cache/8a/stap_8aaac3604a689cbbd6ae1226de2f15db_1317.c
Pass 4: using cached /root/.systemtap/cache/8a/stap_8aaac3604a689cbbd6ae1226de2f15db_1317.ko
Pass 5: starting run.
# ema's note: output stops here

To make sure that nothing basic is wrong with the systemtap setup I looked for icmp_reply, that does print a backtrace.

In T260281#6382529, @ema wrote:

I've installed systemtap on mw1357

Nevermind, I've seen only now that mw1357 is depooled. Here's some preliminary results from mw1359: P12251. Note that the calls happen in bursts but rarely (12:16:30, 12:16:45, 12:17:31). Also note that the traces include calls to perf_trace_run_bpf_submit, it might be wise to run the two different instrumentation tools on different hosts for clarity.

node_vmstat_nr_slab_unreclaimable is going up indefinitely on nodes affected by the issue, following a pattern that matches the general memory usage. However, the actual amount of "lost" memory does not match the size of unreclaimable slabs, which is only ~2G on mw1357:

root@mw1357:~# grep SUnreclaim /proc/meminfo 
SUnreclaim:      2097548 kB

We discussed two things to try out next:

• reboot 4.9 with memory accounting disabled (cgroup.memory=nokmem) to see if we ran into https://lwn.net/ml/linux-kernel/20190611231813.3148843-1-guro@fb.com/, in particular considering that mem_cgroup_try_charge is featured often in the memcg_schedule_kmem_cache_create traces that @CDanis suggested to gather (see P12251 and P12252)
• reboot with 4.19 from stretch-backports because it's a low hanging fruit and 4.9 is very old

Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:38:52Z] <ema> reboot mw1382 with kernel memory accounting disabled T260281

Mentioned in SAL (#wikimedia-operations) [2020-08-13T14:45:55Z] <ema> repool mw1382 with kernel memory accounting disabled T260281

I suggest setting this a security issue since this may cause people to intentionally make memory leaks to damage servers using this software.

In T260281#6385334, @NullPointer wrote:

I suggest setting this a security issue since this may cause people to intentionally make memory leaks to damage servers using this software.

If it's related to Score/Lillypond as speculated, then no third party sight should be running it anyway due to the security issues found,

Script wmf-auto-reimage was launched by cdanis on cumin1001.eqiad.wmnet for hosts:

mw1359.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202008171607_cdanis_15670_mw1359_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['mw1359.eqiad.wmnet']

and were ALL successful.

We're tracking this, but unsure as to next steps. Let us know if more active investigation from Platform team is needed.