| fnegri | |
| May 16 2024, 2:40 PM |
| F53500549: Screenshot 2024-05-16 at 17.48.28.png | |
| May 16 2024, 4:04 PM |
| F53492686: Screenshot 2024-05-16 at 16.25.13.png | |
| May 16 2024, 2:40 PM |
Many clouddb* hosts show a pattern where the free memory decreases very slowly over time. This triggers a warning alert when the free memory is lower than 5%.
I wonder if this could be something similar to T353093: [toolsdb] MariaDB process is killed by OOM killer (December 2023) (although the decrease rate there was much faster), and was fixed by switching to jemalloc.
Mentioned in SAL (#wikimedia-operations) [2024-05-16T15:45:12Z] <dhinus> systemctl restart mariadb@s4.service on clouddb1015 (using too much RAM) T365164
Restarting the mariadb@s4.service freed up about 300G of RAM:
I've added a note on this procedure to the alert runbook.
I also found a SAL log that indicates a similar thing happened on the same host a few months ago: https://sal.toolforge.org/log/wbX_F40BGiVuUzOdbGXw
New memory alerts started triggering for clouddb1016 and clouddb1020. I will restart them on Monday.
I did not restart the services, but the alerts disappeared from alerts.wikimedia.org. I can see they are still in status WARNING in Icinga though, I'm not sure why they are no longer visible in alerts.wikimedia.org.