https://grafana.wikimedia.org/d/XyoE_N_Wz/wikidata-database-cpu-saturation?orgId=1 shows the values for eqiad but there is no equivalent for codfw. This was called out during the Sep 2020 DC failover.
Description
Description
Event Timeline
Comment Actions
To give some context to why we have an specific CPU one for s8 (wikidatawiki): a few months ago wikidatawiki was having serious performance issues, when we compressed InnoDB (and we had less servers), so we had to tweak LB weights mostly based on CPU usage.
Migrating to 10.4 showed a massive improvements when dealing with InnoDB compression and also we've added more servers to that section, so we are out of the woods in that sense.
But it is definitely useful to have a codfw CPU dashboard for s8 as well, at least to mimic what we have in eqiad.
Comment Actions
I am not sure if we should keep this open anymore. We are not having CPU usage problems in s8 anymore (we have most of hosts running 10.4 and we have lots of hosts already).
If we do want to keep the codfw CPU dashboard, we'd also need to make sure it is populated automatically, as the eqiad one is done manually, that is: we have to add/remove new/old hosts from that dashboard manually.
With the last server movements in s8, I am pretty sure that eqiad dashboard doesn't reflect the reality anymore.
Comment Actions
Dropping priority for now. I'll poke around the dashboards, see if I have any ideas and close if I won't.
Comment Actions
I am going to close this as https://grafana.wikimedia.org/d/XyoE_N_Wz/wikidata-database-cpu-saturation?orgId=1 isn't maintained (it is not updated with the number of hosts) and I don't think we've needed this anymore since we had the issues 2 years ago.
Reopen if you believe it still needs to be worked on!