In an attempt to fix https://phabricator.wikimedia.org/T436663, we removed the temporary mediawiki-multiversion-cli image pinning we put in place a couple of weeks ago as configuration drift caused a influx of mediawiki errors, confusing deployers and putting stress on our logging infrastructure.
We've reverted that pin, thus reactivating the mechanism causing us to always use the last mediaiwki image (currently docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-cli:2026-09-02-080827-publish-83). And the OOMs have now re-appeared. Example: https://airflow-dumps.wikimedia.org/dags/mediawiki_dumps_sql_xml_large_a_to_z_full/grid?dag_run_id=scheduled__2026-08-01T08%3A00%3A00%2B00%3A00&task_id=jawiki.dump_remaining_full&tab=logs
...
'container_statuses': [{'allocated_resources': None,
'allocated_resources_status': None,
'container_id': 'containerd://561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e',
'image': 'docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-cli:2026-09-02-080827-publish-83',
'image_id': 'docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-cli@sha256:d082436593c1ef25d0284b300957a7fb7693b4654651a08d71834a72d7608d0c',
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': 'mediawiki-dump-sql-xml',
'ready': False,
'resources': None,
'restart_count': 0,
'started': False,
'state': {'running': None,
'terminated': {'container_id': 'containerd://561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e',
'exit_code': 137,
'finished_at': datetime.datetime(2026, 9, 3, 2, 20, 58, tzinfo=tzlocal()),
'message': None,
'reason': 'OOMKilled', # <---- HERE
'signal': None,
'started_at': datetime.datetime(2026, 9, 2, 14, 54, 42, tzinfo=tzlocal())},
'waiting': None},
'stop_signal': None,
'user': None,
'volume_mounts': [{'mount_path': '/etc/wikimedia-cluster',
'name': 'mediawiki-production-wikimedia-cluster',
'read_only': None,
'recursive_read_only': None},
{'mount_path': '/run/shared',
'name': 'shared-socket',
'read_only': None,
'recursive_read_only': None},
{'mount_path': '/etc/wmerrors',
'name': 'mediawiki-production-wmerrors',
'read_only': None,
'recursive_read_only': None},
{'mount_path': '/var/log/php-fpm',
'name': 'php-logging',
'read_only': None,
...Looking at a before / after situation, we do indeed see that there's a notable increase of memory use, going from peak 8GB to peak 13GB.
| Before (Buster) | After (Bookworm) |
|---|---|
| {F101630388} | {F101630387} |
We find an OOM report in the kernel logs:
[4229208.860721] lbzip2 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991 [4229208.860732] CPU: 113 UID: 33 PID: 2588786 Comm: lbzip2 Not tainted 6.12.88+deb12-amd64 #1 Debian 6.12.88-1~bpo12+1 [4229208.860736] Hardware name: Supermicro SYS-112C-TN/X14SBHM, BIOS 1.5 12/12/2025 [4229208.860737] Call Trace: [4229208.860741] <TASK> [4229208.860747] dump_stack_lvl+0x64/0x80 [4229208.860756] dump_header+0x43/0x1d0 [4229208.860761] oom_kill_process+0xfa/0x200 [4229208.860763] out_of_memory+0x257/0x520 [4229208.860766] mem_cgroup_out_of_memory+0x12a/0x140 [4229208.860773] try_charge_memcg+0x491/0x620 [4229208.860777] __mem_cgroup_charge+0x42/0xd0 [4229208.860780] do_anonymous_page+0x2db/0x7d0 [4229208.860786] ? __pte_offset_map+0x1b/0x180 [4229208.860791] __handle_mm_fault+0xb52/0xf90 [4229208.860795] handle_mm_fault+0x180/0x2d0 [4229208.860798] do_user_addr_fault+0x20c/0x670 [4229208.860802] exc_page_fault+0x76/0x190 [4229208.860808] asm_exc_page_fault+0x26/0x30 [4229208.860814] RIP: 0033:0x55eda0d85834 [4229208.860819] Code: 5f ff 85 ff 0f 84 9c 03 00 00 c1 ea 08 45 8d 68 ff 41 8b 14 94 89 d3 45 85 c0 0f 84 22 01 00 00 48 89 4c 24 08 89 c5 c1 e8 18 <88> 1e 31 d0 c1 e5 08 0f b6 c0 41 33 2c 81 89 e8 44 38 d3 0f 84 63 [4229208.860821] RSP: 002b:00007fa33f252e00 EFLAGS: 00010206 [4229208.860823] RAX: 0000000000000011 RBX: 0000000003bd0769 RCX: 00000000000b7cd2 [4229208.860824] RDX: 0000000003bd0769 RSI: 00007fa3388c9000 RDI: 00000000000b7998 [4229208.860825] RBP: 00000000113fe0e0 R08: 00000000000b7cd0 R09: 000055eda0da1020 [4229208.860826] R10: 0000000004144064 R11: 00000000000b7997 R12: 00007fa33800fb30 [4229208.860827] R13: 00000000000b7ccf R14: 0000000003796861 R15: 00007fa3388a5110 [4229208.860830] </TASK> [4229208.860721] lbzip2 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991 [4229208.860732] CPU: 113 UID: 33 PID: 2588786 Comm: lbzip2 Not tainted 6.12.88+deb12-amd64 #1 Debian 6.12.88-1~bpo12+1 [4229208.860736] Hardware name: Supermicro SYS-112C-TN/X14SBHM, BIOS 1.5 12/12/2025 [4229208.860737] Call Trace: [4229208.860741] <TASK> [4229208.860747] dump_stack_lvl+0x64/0x80 [4229208.860756] dump_header+0x43/0x1d0 [4229208.860761] oom_kill_process+0xfa/0x200 [4229208.860763] out_of_memory+0x257/0x520 [4229208.860766] mem_cgroup_out_of_memory+0x12a/0x140 [4229208.860773] try_charge_memcg+0x491/0x620 [4229208.860777] __mem_cgroup_charge+0x42/0xd0 [4229208.860780] do_anonymous_page+0x2db/0x7d0 [4229208.860786] ? __pte_offset_map+0x1b/0x180 [4229208.860791] __handle_mm_fault+0xb52/0xf90 [4229208.860795] handle_mm_fault+0x180/0x2d0 [4229208.860798] do_user_addr_fault+0x20c/0x670 [4229208.860802] exc_page_fault+0x76/0x190 [4229208.860808] asm_exc_page_fault+0x26/0x30 [4229208.860814] RIP: 0033:0x55eda0d85834 [4229208.860819] Code: 5f ff 85 ff 0f 84 9c 03 00 00 c1 ea 08 45 8d 68 ff 41 8b 14 94 89 d3 45 85 c0 0f 84 22 01 00 00 48 89 4c 24 08 89 c5 c1 e8 18 <88> 1e 31 d0 c1 e5 08 0f b6 c0 41 33 2c 81 89 e8 44 38 d3 0f 84 63 [4229208.860821] RSP: 002b:00007fa33f252e00 EFLAGS: 00010206 [4229208.860823] RAX: 0000000000000011 RBX: 0000000003bd0769 RCX: 00000000000b7cd2 [4229208.860824] RDX: 0000000003bd0769 RSI: 00007fa3388c9000 RDI: 00000000000b7998 [4229208.860825] RBP: 00000000113fe0e0 R08: 00000000000b7cd0 R09: 000055eda0da1020 [4229208.860826] R10: 0000000004144064 R11: 00000000000b7997 R12: 00007fa33800fb30 [4229208.860827] R13: 00000000000b7ccf R14: 0000000003796861 R15: 00007fa3388a5110 [4229208.860830] </TASK> [4229208.860831] memory: usage 16777216kB, limit 16777216kB, failcnt 13120558 [4229208.860834] swap: usage 0kB, limit 0kB, failcnt 0 [4229208.860835] Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb83ded_48c7_4f7b_ba70_5d0e4019b42a.slice/cri-containerd-561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e.scope: [4229208.860951] anon 17101406208 [4229208.860953] file 4280320 [4229208.860953] kernel 74182656 [4229208.860954] kernel_stack 4079616 [4229208.860955] pagetables 36864000 [4229208.860955] sec_pagetables 0 [4229208.860956] percpu 46280 [4229208.860956] sock 0 [4229208.860957] vmalloc 8192 [4229208.860957] shmem 0 [4229208.860958] zswap 0 [4229208.860959] zswapped 0 [4229208.860959] file_mapped 0 [4229208.860960] file_dirty 4235264 [4229208.860960] file_writeback 0 [4229208.860961] swapcached 0 [4229208.860961] anon_thp 16940793856 [4229208.860962] file_thp 0 [4229208.860963] shmem_thp 0 [4229208.860963] inactive_anon 5088321536 [4229208.860964] active_anon 12013056000 [4229208.860964] inactive_file 4087808 [4229208.860965] active_file 188416 [4229208.860965] unevictable 0 [4229208.860966] slab_reclaimable 29598760 [4229208.860967] slab_unreclaimable 3131184 [4229208.860967] slab 32729944 [4229208.860968] workingset_refault_anon 0 [4229208.860969] workingset_refault_file 2088340 [4229208.860969] workingset_activate_anon 0 [4229208.860970] workingset_activate_file 2006260 [4229208.860970] workingset_restore_anon 0 [4229208.860971] workingset_restore_file 66 [4229208.860971] workingset_nodereclaim 533637 [4229208.860972] pgdemote_kswapd 0 [4229208.860973] pgdemote_direct 0 [4229208.860973] pgdemote_khugepaged 0 [4229208.860974] pgpromote_success 0 [4229208.860974] pgscan 94780244 [4229208.860975] pgsteal 94779286 [4229208.860976] pgscan_kswapd 0 [4229208.860976] pgscan_direct 94752532 [4229208.860977] pgscan_khugepaged 27712 [4229208.860978] pgsteal_kswapd 0 [4229208.860978] pgsteal_direct 94751574 [4229208.860979] pgsteal_khugepaged 27712 [4229208.860979] pgfault 47285069 [4229208.860980] pgmajfault 0 [4229208.860980] pgrefill 4459591 [4229208.860981] pgactivate 0 [4229208.860981] pgdeactivate 0 [4229208.860982] pglazyfree 0 [4229208.860982] pglazyfreed 0 [4229208.860983] swpin_zero 0 [4229208.860983] swpout_zero 0 [4229208.860984] zswpin 0 [4229208.860984] zswpout 0 [4229208.860985] zswpwb 0 [4229208.860986] thp_fault_alloc 243048 [4229208.860986] thp_collapse_alloc 441 [4229208.860987] thp_swpout 0 [4229208.860987] thp_swpout_fallback 0 [4229208.860988] numa_pages_migrated 17620271 [4229208.860988] numa_pte_updates 115593610 [4229208.860989] numa_hint_faults 18075375 [4229208.860990] Tasks state (memory values in pages): [4229208.860990] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytesswapents oom_score_adj name [4229208.860992] [ 59843] 33 59843 1154 530 0 530 0 57344 0 991 worker [4229208.860997] [ 59870] 33 59870 159573 6249 3902 2347 0 249856 0 991 python3 [4229208.861000] [2588736] 33 2588736 646 268 0 268 0 45056 0 991 sh [4229208.861003] [2588737] 33 2588737 55952 2343 2079 264 0 86016 0 991 lbzip2 [4229208.861006] [2588738] 33 2588738 941922 575726 574674 1052 0 5001216 0 991 7za [4229208.861009] [2588739] 33 2588739 646 269 0 269 0 40960 0 991 sh [4229208.861011] [2588742] 33 2588742 55952 2571 2305 266 0 81920 0 991 lbzip2 [4229208.861013] [2588743] 33 2588743 1077898 698700 697646 1054 0 6008832 0 991 7za [4229208.861016] [2588744] 33 2588744 646 261 0 261 0 49152 0 991 sh [4229208.861018] [2588746] 33 2588746 55952 2542 2281 261 0 86016 0 991 lbzip2 [4229208.861020] [2588747] 33 2588747 1140464 739626 738570 1056 0 6356992 0 991 7za [4229208.861023] [2588748] 33 2588748 646 264 0 264 0 40960 0 991 sh [4229208.861025] [2588753] 33 2588753 55952 2284 2025 259 0 86016 0 991 lbzip2 [4229208.861027] [2588754] 33 2588754 1290681 831649 830595 1054 0 7147520 0 991 7za [4229208.861029] [2588758] 33 2588758 646 260 0 260 0 40960 0 991 sh [4229208.861031] [2588759] 33 2588759 55952 1829 1563 266 0 98304 0 991 lbzip2 [4229208.861033] [2588762] 33 2588762 996292 606475 605691 784 0 5226496 0 991 7za [4229208.861035] [2588763] 33 2588763 646 259 0 259 0 45056 0 991 sh [4229208.861037] [2588765] 33 2588765 55952 2550 2281 269 0 86016 0 991 lbzip2 [4229208.861070] [2588767] 33 2588767 1084045 698879 697823 1056 0 6029312 0 991 7za [4229208.861072] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb83ded_48c7_4f7b_ba70_5d0e4019b42a.slice/cri-containerd-561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb83ded_48c7_4f7b_ba70_5d0e4019b42a.slice/cri-containerd-561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e.scope,task=7za,pid=2588754,uid=33 [4229208.861192] Memory cgroup out of memory: Killed process 2588754 (7za) total-vm:5162724kB, anon-rss:3322380kB, file-rss:4216kB, shmem-rss:0kB, UID:33 pgtables:6980kB oom_score_adj:991 [4229208.880147] Tasks in /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb83ded_48c7_4f7b_ba70_5d0e4019b42a.slice/cri-containerd-561e3635c9457aa7d9727a7ef67016cd2761cf7e5ad5f2fbf7ec33830d72046e.scope are going to be killed due to memory.oom.group set [4229208.880157] Memory cgroup out of memory: Killed process 59843 (worker) total-vm:4616kB, anon-rss:0kB, file-rss:2120kB, shmem-rss:0kB, UID:33 pgtables:56kB oom_score_adj:991 [4229208.897958] Memory cgroup out of memory: Killed process 59912 (python3) total-vm:638292kB, anon-rss:15608kB, file-rss:9388kB, shmem-rss:0kB, UID:33 pgtables:244kB oom_score_adj:991 [4229208.959620] Memory cgroup out of memory: OOM victim 2588737 (lbzip2) is already exiting. Skip killing the task [4229208.959626] Memory cgroup out of memory: OOM victim 2588738 (7za) is already exiting. Skip killing the task [4229208.959636] Memory cgroup out of memory: Killed process 2588739 (sh) total-vm:2584kB, anon-rss:0kB, file-rss:1076kB, shmem-rss:0kB, UID:33 pgtables:40kB oom_score_adj:991 [4229209.159873] Memory cgroup out of memory: Killed process 2588744 (sh) total-vm:2584kB, anon-rss:0kB, file-rss:1044kB, shmem-rss:0kB, UID:33 pgtables:48kB oom_score_adj:991 [4229209.177553] Memory cgroup out of memory: Killed process 2588786 (lbzip2) total-vm:223808kB, anon-rss:8100kB, file-rss:1036kB, shmem-rss:0kB, UID:33 pgtables:84kB oom_score_adj:991`
The important bits are
82 [4229208.860721] lbzip2 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=991
[4229208.860831] memory: usage 16777216kB, limit 16777216kB, failcnt 13120558
[4229208.861192] Memory cgroup out of memory: Killed process 2588754 (7za) total-vm:5162724kB, anon-rss:3322380kB, file-rss:4216kB, shmem-rss:0kB, UID:33 pgtables:6980kB oom_score_adj:991
In the buster image, I see the following 7za versions being installed
www-data@dse-k8s-worker1020:/$ dpkg -l | grep 7z ii p7zip 16.02+dfsg-8 amd64 7zr file archiver with high compression ratio ii p7zip-full 16.02+dfsg-8 amd64 7z and 7za file archivers with high compression ratio
and the following in the bookworm image
www-data@dse-k8s-worker1020:/tmp$ dpkg -l | grep 7z ii p7zip 16.02+really26.02+dfsg-0+deb12u1 amd64 7zr file archiver with high compression ratio ii p7zip-full 16.02+really26.02+dfsg-0+deb12u1 amd64 7z and 7za file archivers with high compression ratio
Assuming that 7za is using a calendar versioning scheme, that 10 years of diff.
I've tried to understand whether 7za had changed behavior, and if we needed to pass a specific flag to limit the amount of threads it uses. So far, I'm not seeing any change in behavior.
I'm creating a large local file using
dd if=/dev/zero of=/tmp/bigfile bs=1M count=4096
and compressing it in both the buster and bookworm containers using
www-data@dse-k8s-worker1020:/tmp$ time 7za a /tmp/bigfile.bz2 /tmp/bigfile -tbzip2
while looking at the number of threads and peak memory using
watch -n 0.2 'grep -E "VmRSS|VmPeak|Threads" /proc/$(pgrep 7za)/status'
In both cases, I'm getting something akin to
VmPeak: 1478704 kB VmRSS: 732072 kB Threads: 65
The number of threads didn't dramatically shoot up between versions, which could have explained the memory increase.
At this point, and given that we're planning to deprecate this part of the dumps, we might just need to increase our cgroup memory limits?