Via Alertmanager:
WARN Memory 95% used. Largest process: mysqld (1866555) = 73.8%
This is a task to bring down the InnoDB buffer pool size to reduce memory warnings and experiment with whether this causes no material query degradation. If a change here causes material query degradation or alerts begin to resurface again later, the change would need to be reverted and alternative root cause analysis (e.g., for memory leaks) and exploration of mitigations would need to be performed.
dr0ptp4kt@clouddb1019:~$ top -b -n 1 -o +%MEM | head -10 | tail -4
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1866555 mysql 20 0 377.0g 371.1g 13984 S 18.8 73.8 66036:54 mysqld
1866396 mysql 20 0 107.5g 102.6g 13836 S 0.0 20.4 6757:55 mysqld
502617 root 20 0 171208 79248 78068 S 0.0 0.0 89:38.21 systemd-journalFor now the host seems stable, but this is being filed in anticipation of possible treatment next week.