This happened twice (by the irc logs) in less than one day:
```
taavi> i'm looking at toolsdb
01:53:57 oom killed once again
01:53:58 [3211718.996904] Out of memory: Killed process 725680 (mysqld) total-vm:64552308kB, anon-rss:63683220kB, file-rss:0kB, shmem-rss:0kB, UID:497 pgtables:125676kB oom_score_adj:-600
01:55:03 last one was on nov 9th, it made it almost a month without crashing
01:55:11 <andrewbogott> oops I just rebooted it out from under you sorry
01:55:26 But I imagine you were about to do that anyway :)
01:55:56 taavi: do you mind setting r/w after it comes up? I have food on the stove that's about to burn :)
01:56:00
<taavi> sure
01:56:18 yeah I was just going through lots of email from the past week and happened to spot that particular one just as it got in
01:56:22 <andrewbogott> thank you! I'll check back in a few to make sure it was an easy resolution
01:58:24
<taavi> mariadb is taking suspiciously long to even start
01:58:54 there we go
08:14:07 — dcaro paged
08:16:57 <dcaro> oom killed too
08:16:57 Dec 09 07:03:41 tools-db-1 systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
08:18:29 I'm starting it again
08:19:32 made it writable again
```
```
root@tools-db-1:~# journalctl | grep oom-kill
Oct 29 17:23:09 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600...
Oct 29 17:23:13 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=1946908,uid=497Dec 09 00:45:39 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600
Oct 30 17:32:29 tools-db-1 kernel: prometheus-node invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=Dec 09 00:45:39 tools-db-1 kernel: oom_kill_process.cold+0xb/0x10
Oct 30 17:32:29 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=2498,uid=497Dec 09 00:45:39 tools-db-1 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 30 17:32:30 tools-db-1 systemd[1]: mariadb.service: Failed with result 'oom-kill'.Dec 09 00:45:39 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=725680,uid=497
Oct 31 17:27:00 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP)Dec 09 00:45:39 tools-db-1 kernel: Out of memory: Killed process 725680 (mysqld) total-vm:64552308kB, anon-rss:63683220kB, file-rss:0kB, order=0shmem-rss:0kB, UID:497 pgtables:125676kB oom_score_adj=:-600
Oct 31 17:27:00 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=2309,uid=497Dec 09 00:45:40 tools-db-1 kernel: oom_reaper: reaped process 725680 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Oct 31 17:27:01 tools-db-1 systemd[1]: mariadb.service: Failed with result 'oom-kill'.Dec 09 07:03:41 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600
Nov 01 20:29:33 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-60Dec 09 07:03:41 tools-db-1 kernel: oom_kill_process.cold+0xb/0x10
Nov 01 20:29:33 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=92611,uid=497Dec 09 07:03:41 tools-db-1 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Nov 03 05:34:07 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600Dec 09 07:03:41 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=892,uid=497
Nov 03 05:34:08 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=909,uid=497Dec 09 07:03:41 tools-db-1 kernel: Out of memory: Killed process 892 (mysqld) total-vm:68657680kB, anon-rss:64977180kB, file-rss:0kB, shmem-rss:0kB, UID:497 pgtables:129096kB oom_score_adj:-600
Nov 03 05:34:08Dec 09 07:03:41 tools-db-1 systemd[1]: mariadb.service: Failed with result 'oom-A process of this unit has been killed by the OOM kill'er.
Nov 08 11:08:54Dec 09 07:03:42 tools-db-1 kernel: mysqld invoked oom-kill_reaper: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP)reaped process 892 (mysqld), now anon-rss:0kB, order=0file-rss:0kB, oom_score_adj=-600shmem-rss:0kB
Nov 08 11:08:54 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=119707,uid=497Dec 09 07:03:43 tools-db-1 systemd[1]: mariadb.service: Failed with result 'oom-kill'.
NovDec 09 11:12:2023:37 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600
Nov 09 11:12:20 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=575274,uid=497Dec 09 11:23:37 tools-db-1 kernel: oom_kill_process.cold+0xb/0x10
Nov 10 03:53:04 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600Dec 09 11:23:37 tools-db-1 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Nov 10 03:53:04Dec 09 11:23:37 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=66397826159,uid=497
# todayDec 09 11:23:37 tools-db-1 kernel: Out of memory: Killed process 26159 (mysqld) total-vm:67846824kB, anon-rss:64994320kB, file-rss:0kB, shmem-rss:0kB, UID:497 pgtables:129200kB oom_score_adj:-600
Dec 09 00:4511:23:39 tools-db-1 kernel: mysqld invoked oom-kill_reaper: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP)reaped process 26159 (mysqld), now anon-rss:0kB, order=0file-rss:0kB, oom_score_adj=-600shmem-rss:0kB
Dec 09 00:45:39 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=725680,uid=49716:11:12 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-600
Dec 09 07:03:41 tools-db-1 kernel: mysqld invoked oom-killer: gfp_mask=0x1140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=-6016:11:12 tools-db-1 kernel: oom_kill_process.cold+0xb/0x10
Dec 09 07:03:41 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=892,uid=49716:11:12 tools-db-1 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Dec 09 07:03:43 tools-db-1 systemd[1]: mariadb.service: Failed with result 'oom-kill'.16:11:12 tools-db-1 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=41033,uid=497
Dec 09 16:11:12 tools-db-1 kernel: Out of memory: Killed process 41033 (mysqld) total-vm:68536228kB, anon-rss:64961000kB, file-rss:0kB, shmem-rss:0kB, UID:497 pgtables:129092kB oom_score_adj:-600
Dec 09 16:11:14 tools-db-1 kernel: oom_reaper: reaped process 41033 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
```
It had been a while, getting "stabler" :)