Page MenuHomePhabricator

Power Consumption Not Showing Up in Graphs for Machine Learning PDUs
Closed, ResolvedPublic

Description

Hi Observability Team - we have some PDUs for the Machine Learning GPU servers in racks E9, E10, E11, E12, E13 and E14 at EQIAD that aren't showing up on the following link with their power consumption numbers:

https://grafana-rw.wikimedia.org/d/f64mmDzMz/power-dashboard?orgId=1&from=now-30d&to=now&forceLogin=true&timezone=utc&var-datasource=PA7DE9A562EF40E24&var-site=eqiad&var-instance=$__all&var-row=$__all

The PDU device type being used are listed as Sentry C2S36TE-DQME2M99 and CSL36TE-DQME2M99. Can you look into why the data isn't coming up in Grafana?

Thanks,
Willy

Event Timeline

Change #1345124 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] hiera/pdu_families: update regexps to match new models

https://gerrit.wikimedia.org/r/1345124

Hi @wiki_willy

I'm unable to find devices of such a type on netbox: CSL36TE-DQME2M99. Is there a typo?

For the other one, I've just pushed a patch that adjusts the regexp used to match PDU models.

I'm unable to find the product code (C2L36TE-DQME2M99) on the Server Technology product selector. If I've correctly understood the naming convention, it should be a three-phase PDU. Can you confirm that?

The patch I posted on Friday should already match this model as a three-phase PDU.

Thanks!

Hi @tappof - that's correct, it's 3-phase. I'm not sure why it's not showing up on the ServerTech site, but here's the quote from one of the orders that we placed: T382370

Change #1345124 merged by Tiziano Fogli:

[operations/puppet@production] hiera/pdu_families: update regexps to match new models

https://gerrit.wikimedia.org/r/1345124

Thanks @wiki_willy,

I've just submitted the patch. Let me know if you notice something weird.

Thanks @tappof! Looks like they're showing up now