As per parent task, Prometheus k8s in codfw/eqiad can get OOM killed from time to time.
As one of the short term mitigations I think we should try to scale the hw up vertically, namely by increasing memory.
The current R440 hosts have 4x32GB RAM each, @wiki_willy would you mind helping in looking if we have memory available on site to be installed? Ideally >= 64GB per host (total 4 hosts, 2 eqiad and 2 codfw).
And if not immediately available, could we order the memory?
thank you!
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T354399 Prometheus @ k8s OOM loop | |||
| Resolved | fgiunchedi | T354606 Investigate memory increase for Prometheus hosts in codfw/eqiad | |||
| Resolved | VRiley-WMF | T354684 RAM upgrade for prometheus100[56] | |||
| Resolved | Jhancock.wm | T354685 RAM upgrade for prometheus200[56] |
@Papaul / @Jhancock.wm and @Jclark-ctr / @VRiley-WMF - can you see if you have any spare memory onsite for Filippo? I think it's for prometheus100[5,6] and prometheus200[5,6]. (cc @RobH in case we have to order them)
https://netbox.wikimedia.org/search/?q=prometheus&obj_type=
Thanks,
Willy
Awesome, thanks @Jhancock.wm. Here's the codfw upgrade ticket for you to coordinate with @fgiunchedi on the downtime - T354685. Thanks, Willy
Thanks @VRiley-WMF. I have T354684 assigned over to you, so you can work with @fgiunchedi on coordinating downtime for the upgrades. Thanks, Willy