Page MenuHomePhabricator
Create Task
Maniphest T225720

poll power data for redeployment of esams/knams
Closed, ResolvedPublic0 Estimated Story Points
Actions

Description

This task will track the power data usage figures for all the hardware we plan to re-deploy in esams/knams. This data is needed so we know what our actual power footprint requirement is per rack.

The power load was polled via 'show chassis power' but is NOT historic and these may not be the peak values. We'll need @ayounsi to confirm these and let us konw what the peak hours of usage is for each router, since we want peak power use. (Or perhaps there is a command to pull the peak power use that I am unaware of? Not according to @ayounsi when asked by @RobH)

power figures for 2019-07-20

@RobH pulled the data in comments below, summarized here:

hostnamedevice typewattage
cr1-eqiadMX4801488 W
cr2-eqiadMX4801425 W
asw2-b2-eqiadQFX510048 W
asw2-b4-eqiadQFX510048 W
asw2-b7-eqiadQFX510048 W
asw2-d2-eqiadQFX510048 W
asw2-d4-eqiadQFX510048 W
asw2-d7-eqiadQFX510048 W
cp1075R440 (cp text)198 W
cp1081R440 (cp text)220 W
cp1087R440 (cp text)198 W
cp1076R440 (cp upload)198 W
cp1082R440 (cp upload)198 W
cp1088R440 (cp upload)176 W

Event Timeline

RobH triaged this task as Medium priority.Jun 13 2019, 12:54 PM
RobH created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2019, 12:54 PM
RobH added a comment.EditedJun 13 2019, 12:56 PM
5 $> ssh cr2-esams.wikimedia.org
--- JUNOS 13.3R8.7 built 2015-10-23 21:23:16 UTC
{master}
robh@re0.cr2-esams> show power
                         ^
syntax error, expecting <command>.

{master}
robh@re0.cr2-esams> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 290 W (zone 0, 5 A at 58 V, 14% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 57 W (zone 0, 1 A at 57 V, 2% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 114 W (zone 0, 2 A at 57 V, 5% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 228 W (zone 0, 4 A at 57 V, 11% of capacity)

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   1087 W (3013 W remaining)
      Actual usage:      689 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 3013 W
4 $> ssh cr1-codfw.wikimedia.org
--- JUNOS 13.3R8.7 built 2015-10-23 21:23:16 UTC
robh@re0.cr1-codfw> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

System:
  Zone 0:
      Capacity:          3180 W (maximum 3180 W)
      Allocated power:   1490 W (1690 W remaining)
      Actual usage:      800 W
  Total system capacity: 3180 W (maximum 3180 W)
  Total remaining power: 1690 W

robh@re0.cr1-codfw> exit
5 $> ssh cr2-codfw.wikimedia.org
--- JUNOS 13.3R8.7 built 2015-10-23 21:23:16 UTC
robh@re0.cr2-codfw> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 250 W (zone 0, 5 A at 50 V, 15% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  1590 W (maximum 1590 W)
  DC output: 200 W (zone 0, 4 A at 50 V, 12% of capacity)

System:
  Zone 0:
      Capacity:          3180 W (maximum 3180 W)
      Allocated power:   1490 W (1690 W remaining)
      Actual usage:      850 W
  Total system capacity: 3180 W (maximum 3180 W)
  Total remaining power: 1690 W

robh@re0.cr2-codfw> exit
robh@re0.cr1-eqiad> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 348 W (zone 0, 6 A at 58 V, 16% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 399 W (zone 0, 7 A at 57 V, 19% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 399 W (zone 0, 7 A at 57 V, 19% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   2037 W (2063 W remaining)
      Actual usage:      1488 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 2063 W

robh@re0.cr1-eqiad>
7 $> ssh cr2-eqiad.wikimedia.org
--- JUNOS 13.3R9.13 built 2016-03-01 07:03:30 UTC
robh@re0.cr2-eqiad> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 399 W (zone 0, 7 A at 57 V, 19% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   2037 W (2063 W remaining)
      Actual usage:      1425 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 2063 W

robh@re0.cr2-eqiad>
RobH updated the task description. (Show Details)Jun 13 2019, 1:09 PM
RobH added a subscriber: ayounsi.

My understanding is we won't be using any MX80s when this is all done, so I did not pull that info.

I'm not sure of the peak usage hours for each site, or if there is a juiper command to poll the peak usage for past 24 hours, rather than just the current usage. Perhaps @ayounsi knows?

Also, for @BBlack, can you advise on the peak hours for usage on the CP systems in eqiad? That way I'll poll their power at their peak times.

ayounsi added a comment.Jun 13 2019, 1:17 PM

I don't think Junos have that feature.

You can find the peak time of a device using their "overall traffic" graph in LibreNMS (eg. https://librenms.wikimedia.org/device/device=145/ )

• ema moved this task from Backlog to Hardware on the Traffic board.Jun 17 2019, 9:15 AM
RobH added a comment.EditedJun 20 2019, 4:30 PM

Ok, in checking, EQIAD seems to enter its PEAK usage around 20:00 GMT (so about a half an hour from now at 10:00 Pacific.)

I'll pull both the 'show chassis power' on cr1 and cr2-eqiad at that time, as well as poll the power from 3 of the new cp systems per group (text/upload): cp1075+ are the new R440 range of cp systems, with the below in site.pp. So with that, I'll poll text: cp1075, cp1081, cp1087 & upload: cp1076, cp1082, cp1088.

node /^cp10(7[579]|8[13579])\.eqiad\.wmnet$/ {
    interface::add_ip6_mapped { 'main': }
    role(cache::text)
}

node /^cp10(7[68]|8[02468]|90)\.eqiad\.wmnet$/ {
    interface::add_ip6_mapped { 'main': }
    role(cache::upload)
}

There isn't a way to historically poll power data like this, so I'll do this at exactly 10:00.

Edit addition: Also I'll poll the QFX5100 asw2-b7-eqiad (cp1075 switch), and asw2-d2-eqiad (cp1087 switch) to get the peak power use for those switches (which would also be used in caching sites).

RobH added a comment.Jun 20 2019, 4:52 PM

commands to run:

juniper: show chassis power
dell (via idrac): racadm getsensorinfo

RobH added a comment.EditedJun 20 2019, 5:06 PM

Power data:

Power drawn live @ approximately 10:00-10:10 AM Pacific:

cr1-eqiad:

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   2037 W (2063 W remaining)
      Actual usage:      1488 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 2063 W

cr2-eqiad:

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   2037 W (2063 W remaining)
      Actual usage:      1425 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 2063 W

text cp systems:

cp1075:

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      220Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

cp1081:

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      220Watts NA   638Watts 0Watts [N]      594Watts [Y]

cp1087:

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      198Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

upload cp systems:

cp1076:

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      176Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

cp1082:

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      198Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

cp1088:

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      176Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]
RobH added a subscriber: Papaul.Jun 20 2019, 5:14 PM

so for the QFX5100 (thanks @Papaul) the command is:

show chassis environment pem

The 10G switches are PEM 2/4/7, so I'll just include them all:

asw2-b-eqiad:

FPC 2 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   45 degrees C / 113 degrees F       
  Temp Sensor 1              OK   44 degrees C / 111 degrees F       
  Temp Sensor 2              OK   36 degrees C / 96 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3840 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 2 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   42 degrees C / 107 degrees F       
  Temp Sensor 1              OK   40 degrees C / 104 degrees F       
  Temp Sensor 2              OK   34 degrees C / 93 degrees F        
  Fan 0                      4200 RPM                                
  Fan 1                      3900 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
FPC 4 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   40 degrees C / 104 degrees F       
  Temp Sensor 1              OK   40 degrees C / 104 degrees F       
  Temp Sensor 2              OK   29 degrees C / 84 degrees F        
  Fan 0                      4200 RPM                                
  Fan 1                      4020 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 4 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   40 degrees C / 104 degrees F       
  Temp Sensor 1              OK   36 degrees C / 96 degrees F        
  Temp Sensor 2              OK   31 degrees C / 87 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3900 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 7 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   44 degrees C / 111 degrees F       
  Temp Sensor 1              OK   39 degrees C / 102 degrees F       
  Temp Sensor 2              OK   37 degrees C / 98 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3900 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 7 PEM 1 status:                     
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   40 degrees C / 104 degrees F       
  Temp Sensor 1              OK   38 degrees C / 100 degrees F       
  Temp Sensor 2              OK   37 degrees C / 98 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3900 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7

asw2-d-eqiad:

FPC 2 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   39 degrees C / 102 degrees F       
  Temp Sensor 2              OK   33 degrees C / 91 degrees F        
  Fan 0                      4260 RPM                                
  Fan 1                      4020 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 2 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   37 degrees C / 98 degrees F        
  Temp Sensor 1              OK   40 degrees C / 104 degrees F       
  Temp Sensor 2              OK   33 degrees C / 91 degrees F        
  Fan 0                      4200 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 4 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 5 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   39 degrees C / 102 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 7 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   35 degrees C / 95 degrees F        
  Temp Sensor 1              OK   37 degrees C / 98 degrees F        
  Temp Sensor 2              OK   31 degrees C / 87 degrees F        
  Fan 0                      4200 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        3             36       5      
FPC 7 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   35 degrees C / 95 degrees F        
  Temp Sensor 1              OK   36 degrees C / 96 degrees F        
  Temp Sensor 2              OK   31 degrees C / 87 degrees F        
  Fan 0                      4260 RPM                                
  Fan 1                      3900 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7
RobH updated the task description. (Show Details)Jun 20 2019, 6:31 PM
RobH updated the task description. (Show Details)
RobH added a comment.Jun 20 2019, 6:36 PM

Updated from irc chat and @BBlack.

Peak eqiad time is actually 01:30 GMT (18:30 Pacific).

For total Linux hosts: 9x lvs/misc/ganeti type nodes, 16x cache nodes. Basically get an approximate idea and fudge factor it upwards a little. We also will have two MX480s and one QFX5100 per rack.

I'll re-poll the data later today and then determine the overall power requirements per rack.

RobH added a comment.Jun 21 2019, 1:31 AM

Ok, as it is now peak hours (according to @BBlack) for eqiad, I'm re-pulling all the power data now. Please note that I'll update the task description AFTER this post (and likely won't update task description summary until Friday AM.)

Power data below.

networking kit power output:

robh@gir:~
3 $> ssh cr1-eqiad.wikimedia.org
--- JUNOS 13.3R9.13 built 2016-03-01 07:03:30 UTC
robh@re0.cr1-eqiad> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 348 W (zone 0, 6 A at 58 V, 16% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 399 W (zone 0, 7 A at 57 V, 19% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 399 W (zone 0, 7 A at 57 V, 19% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   2037 W (2063 W remaining)
      Actual usage:      1488 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 2063 W

robh@re0.cr1-eqiad> exit 

Connection to cr1-eqiad.wikimedia.org closed.
Killed by signal 1.

robh@gir:~
4 $> ssh cr2-eqiad.wikimedia.org
--- JUNOS 13.3R9.13 built 2016-03-01 07:03:30 UTC
robh@re0.cr2-eqiad> show chassis power 

PEM 0:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

PEM 1:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

PEM 2:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 399 W (zone 0, 7 A at 57 V, 19% of capacity)

PEM 3:
  State:     Online
  AC input:  OK (1 feed expected, 1 feed connected)
  Capacity:  2050 W (maximum 2050 W)
  DC output: 342 W (zone 0, 6 A at 57 V, 16% of capacity)

System:
  Zone 0:
      Capacity:          4100 W (maximum 4100 W)
      Allocated power:   2037 W (2063 W remaining)
      Actual usage:      1425 W
  Total system capacity: 4100 W (maximum 4100 W)
  Total remaining power: 2063 W

robh@re0.cr2-eqiad> exit 

Connection to cr2-eqiad.wikimedia.org closed.
Killed by signal 1.

robh@gir:~
5 $> ssh asw2-b-eqiad.mgmt.eqiad.wmnet
--- JUNOS 14.1X53-D46.7 built 2017-11-23 22:06:48 UTC
{master:2}
robh@asw2-b-eqiad> show chassis environment pem 
FPC 1 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 1 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 2 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   46 degrees C / 114 degrees F       
  Temp Sensor 1              OK   45 degrees C / 113 degrees F       
  Temp Sensor 2              OK   37 degrees C / 98 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3840 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 2 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   43 degrees C / 109 degrees F       
  Temp Sensor 1              OK   40 degrees C / 104 degrees F       
  Temp Sensor 2              OK   34 degrees C / 93 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3840 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 3 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   37 degrees C / 98 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 3 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   36 degrees C / 96 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 4 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   39 degrees C / 102 degrees F       
  Temp Sensor 1              OK   40 degrees C / 104 degrees F       
  Temp Sensor 2              OK   29 degrees C / 84 degrees F        
  Fan 0                      4260 RPM                                
  Fan 1                      4020 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 4 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   40 degrees C / 104 degrees F       
  Temp Sensor 1              OK   36 degrees C / 96 degrees F        
  Temp Sensor 2              OK   31 degrees C / 87 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3900 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 5 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   33 degrees C / 91 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 5 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   36 degrees C / 96 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 6 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   36 degrees C / 96 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 6 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   41 degrees C / 105 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 7 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   42 degrees C / 107 degrees F       
  Temp Sensor 1              OK   38 degrees C / 100 degrees F       
  Temp Sensor 2              OK   36 degrees C / 96 degrees F        
  Fan 0                      4080 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 7 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   37 degrees C / 98 degrees F        
  Temp Sensor 1              OK   36 degrees C / 96 degrees F        
  Temp Sensor 2              OK   35 degrees C / 95 degrees F        
  Fan 0                      4140 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 8 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   36 degrees C / 96 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               
FPC 8 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   36 degrees C / 96 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65180 RPM                               

{master:2}
robh@asw2-b-eqiad> exit 

Connection to asw2-b-eqiad.mgmt.eqiad.wmnet closed.
Killed by signal 1.

robh@gir:~
6 $> ssh asw2-d-eqiad.mgmt.eqiad.wmnet
--- JUNOS 14.1X53-D42.3 built 2017-02-14 20:04:49 UTC
{master:2}
robh@asw2-d-eqiad> show chassis environment pem 
FPC 1 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   52 degrees C / 125 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 1 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   40 degrees C / 104 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 2 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   39 degrees C / 102 degrees F       
  Temp Sensor 1              OK   39 degrees C / 102 degrees F       
  Temp Sensor 2              OK   33 degrees C / 91 degrees F        
  Fan 0                      4260 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 2 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   40 degrees C / 104 degrees F       
  Temp Sensor 2              OK   33 degrees C / 91 degrees F        
  Fan 0                      4200 RPM                                
  Fan 1                      4020 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 3 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   37 degrees C / 98 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 3 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 4 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 4 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 5 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   38 degrees C / 100 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 5 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   36 degrees C / 96 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 6 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   25 degrees C / 77 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 6 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   26 degrees C / 78 degrees F        
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 7 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   34 degrees C / 93 degrees F        
  Temp Sensor 1              OK   37 degrees C / 98 degrees F        
  Temp Sensor 2              OK   31 degrees C / 87 degrees F        
  Fan 0                      4260 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        3             36       5      
FPC 7 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   35 degrees C / 95 degrees F        
  Temp Sensor 1              OK   36 degrees C / 96 degrees F        
  Temp Sensor 2              OK   30 degrees C / 86 degrees F        
  Fan 0                      4200 RPM                                
  Fan 1                      3960 RPM                                
  DC Output           Voltage(V) Current(A)  Power(W)  Load(%)
                          12        4             48       7      
FPC 8 PEM 0 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   53 degrees C / 127 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               
FPC 8 PEM 1 status:
  State                      Online
  Airflow                    Back to Front                           
  Temp Sensor 0              OK   48 degrees C / 118 degrees F       
  Temp Sensor 1              OK   0 degrees C / 32 degrees F         
  Temp Sensor 2              OK   2 degrees C / 35 degrees F         
  Fan 0                      458 RPM                                 
  Fan 1                      65192 RPM                               

{master:2}
robh@asw2-d-eqiad>

R440 power details/output:

8 $> ssh root@cp1075.mgmt.eqiad.wmnet
root@cp1075.mgmt.eqiad.wmnet's password: 
/admin1-> racadm getsensorinfo

Sensor Type : POWER
<Sensor Name>                   <Status>                 <Type>         
PS1 Status                      Present                  AC             
PS2 Status                      Present                  AC             

Sensor Type : TEMPERATURE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#CPU1Temp]
CPU1 Temp                     Ok      73C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#CPU2Temp]
CPU2 Temp                     Ok      64C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#SystemBoardInletTemp]
System Board Inlet Temp       Ok      22C      -7C  47C      3C [Y]      43C [Y]

[Key = iDRAC.Embedded.1#SystemBoardExhaustTemp]
System Board Exhaust Temp     Ok      55C      3C   80C      8C [N]      75C [N]



Sensor Type : FAN
<Sensor Name>                   <Status>    <Reading>   <lc>        <uc>        <PWM %>     
System Board Fan1A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan1B              Ok          3720RPM     480RPM      NA          14%         
System Board Fan2A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan2B              Ok          3720RPM     480RPM      NA          14%         
System Board Fan3A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan3B              Ok          3720RPM     480RPM      NA          14%         
System Board Fan4A              Ok          3720RPM     480RPM      NA          14%         
System Board Fan4B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan5A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan5B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan6A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan6B              Ok          3840RPM     480RPM      NA          14%         

Sensor Type : VOLTAGE
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board DIMM PG            Ok          Good                NA          NA          
System Board PS1 PG FAIL        Ok          Good                NA          NA          
System Board PS2 PG FAIL        Ok          Good                NA          NA          
System Board BP1 PG             Ok          Good                NA          NA          
System Board 1.8V SW PG         Ok          Good                NA          NA          
System Board 2.5V SW PG         Ok          Good                NA          NA          
System Board 5V SW PG           Ok          Good                NA          NA          
System Board PVNN SW PG         Ok          Good                NA          NA          
System Board VSB11 SW PG        Ok          Good                NA          NA          
System Board VSBM SW PG         Ok          Good                NA          NA          
System Board 3.3V B PG          Ok          Good                NA          NA          
CPU1 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM012 VPP PG              Ok          Good                NA          NA          
CPU1 MEM012 VTT PG              Ok          Good                NA          NA          
CPU1 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM345 VPP PG              Ok          Good                NA          NA          
CPU1 MEM345 VTT PG              Ok          Good                NA          NA          
CPU1 VCCIO PG                   Ok          Good                NA          NA          
CPU1 VCORE PG                   Ok          Good                NA          NA          
CPU1 FIVR PG                    Ok          Good                NA          NA          
CPU2 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM012 VPP PG              Ok          Good                NA          NA          
CPU2 MEM012 VTT PG              Ok          Good                NA          NA          
CPU2 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM345 VPP PG              Ok          Good                NA          NA          
CPU2 MEM345 VTT PG              Ok          Good                NA          NA          
CPU2 VCCIO PG                   Ok          Good                NA          NA          
CPU2 VCORE PG                   Ok          Good                NA          NA          
CPU2 FIVR PG                    Ok          Good                NA          NA          
PS2 Voltage 2                   Ok          208.00V             NA          NA          
CPU1 VSA PG                     Ok          Good                NA          NA          
CPU2 VSA PG                     Ok          Good                NA          NA          
PS1 Voltage 1                   Ok          208.00V             NA          NA          

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      198Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]


Sensor Type : PROCESSOR
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
CPU1 Status                     Ok          Presence_Detected   NA          NA          
CPU2 Status                     Ok          Presence_Detected   NA          NA          

Sensor Type : MEMORY
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
DIMM A1                         Ok          Presence_Detected   NA          NA          
DIMM A2                         Ok          Presence_Detected   NA          NA          
DIMM A3                         Ok          Presence_Detected   NA          NA          
DIMM A4                         Ok          Presence_Detected   NA          NA          
DIMM A5                         Ok          Presence_Detected   NA          NA          
DIMM A6                         Ok          Presence_Detected   NA          NA          
DIMM A7                         N/A         Absent              NA          NA          
DIMM A8                         N/A         Absent              NA          NA          
DIMM A9                         N/A         Absent              NA          NA          
DIMM A10                        N/A         Absent              NA          NA          
DIMM B1                         Ok          Presence_Detected   NA          NA          
DIMM B2                         Ok          Presence_Detected   NA          NA          
DIMM B3                         Ok          Presence_Detected   NA          NA          
DIMM B4                         Ok          Presence_Detected   NA          NA          
DIMM B5                         Ok          Presence_Detected   NA          NA          
DIMM B6                         Ok          Presence_Detected   NA          NA          

Sensor Type : BATTERY
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board CMOS Battery       Ok          Present             NA          NA          

Sensor Type : PERFORMANCE
<Sensor Name>                   <Status>    <State>        <lc>      <uc>      
System Board Power Optimized    Ok          Not Degraded   NA        NA        

Sensor Type : INTRUSION
<Sensor Name>                   <Intrusion>    <Status>       
System Board Intrusion          Closed         Power ON       

Sensor Type : REDUNDANCY
<Sensor Name>                   <Status>                 <Type>         
System Board PS Redundancy      Full Redundant           PSU            
System Board Fan Redundancy     Full Redundant           Fan            

Sensor Type : SYSTEM PERFORMANCE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#SystemBoardIOUsage]
System Board IO Usage    Ok             0%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardMEMUsage]
System Board MEM Usage   Ok             11%     NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardSYSUsage]
System Board SYS Usage   Ok             23%     NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardCPUUsage]
System Board CPU Usage   Ok             28%     NA  NA    NA  [N]      101% [Y]

/admin1-> exit
CLP Session terminated
Connection to cp1075.mgmt.eqiad.wmnet closed.
Killed by signal 1.

robh@gir:~
9 $> ssh root@cp1076.mgmt.eqiad.wmnet
root@cp1076.mgmt.eqiad.wmnet's password: 
/admin1-> racadm getsensorinfo

Sensor Type : POWER
<Sensor Name>                   <Status>                 <Type>         
PS1 Status                      Present                  AC             
PS2 Status                      Present                  AC             

Sensor Type : TEMPERATURE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#CPU1Temp]
CPU1 Temp                     Ok      75C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#CPU2Temp]
CPU2 Temp                     Ok      60C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#SystemBoardInletTemp]
System Board Inlet Temp       Ok      22C      -7C  47C      3C [Y]      43C [Y]

[Key = iDRAC.Embedded.1#SystemBoardExhaustTemp]
System Board Exhaust Temp     Ok      52C      3C   80C      8C [N]      75C [N]



Sensor Type : FAN
<Sensor Name>                   <Status>    <Reading>   <lc>        <uc>        <PWM %>     
System Board Fan1A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan1B              Ok          3600RPM     480RPM      NA          14%         
System Board Fan2A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan2B              Ok          3720RPM     480RPM      NA          14%         
System Board Fan3A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan3B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan4A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan4B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan5A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan5B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan6A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan6B              Ok          3720RPM     480RPM      NA          14%         

Sensor Type : VOLTAGE
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board DIMM PG            Ok          Good                NA          NA          
System Board PS1 PG FAIL        Ok          Good                NA          NA          
System Board PS2 PG FAIL        Ok          Good                NA          NA          
System Board BP1 PG             Ok          Good                NA          NA          
System Board 1.8V SW PG         Ok          Good                NA          NA          
System Board 2.5V SW PG         Ok          Good                NA          NA          
System Board 5V SW PG           Ok          Good                NA          NA          
System Board PVNN SW PG         Ok          Good                NA          NA          
System Board VSB11 SW PG        Ok          Good                NA          NA          
System Board VSBM SW PG         Ok          Good                NA          NA          
System Board 3.3V B PG          Ok          Good                NA          NA          
CPU1 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM012 VPP PG              Ok          Good                NA          NA          
CPU1 MEM012 VTT PG              Ok          Good                NA          NA          
CPU1 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM345 VPP PG              Ok          Good                NA          NA          
CPU1 MEM345 VTT PG              Ok          Good                NA          NA          
CPU1 VCCIO PG                   Ok          Good                NA          NA          
CPU1 VCORE PG                   Ok          Good                NA          NA          
CPU1 FIVR PG                    Ok          Good                NA          NA          
CPU2 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM012 VPP PG              Ok          Good                NA          NA          
CPU2 MEM012 VTT PG              Ok          Good                NA          NA          
CPU2 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM345 VPP PG              Ok          Good                NA          NA          
CPU2 MEM345 VTT PG              Ok          Good                NA          NA          
CPU2 VCCIO PG                   Ok          Good                NA          NA          
CPU2 VCORE PG                   Ok          Good                NA          NA          
CPU2 FIVR PG                    Ok          Good                NA          NA          
PS2 Voltage 2                   Ok          208.00V             NA          NA          
CPU1 VSA PG                     Ok          Good                NA          NA          
CPU2 VSA PG                     Ok          Good                NA          NA          
PS1 Voltage 1                   Ok          208.00V             NA          NA          

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      198Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]


Sensor Type : PROCESSOR
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
CPU1 Status                     Ok          Presence_Detected   NA          NA          
CPU2 Status                     Ok          Presence_Detected   NA          NA          

Sensor Type : MEMORY
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
DIMM A1                         Ok          Presence_Detected   NA          NA          
DIMM A2                         Ok          Presence_Detected   NA          NA          
DIMM A3                         Ok          Presence_Detected   NA          NA          
DIMM A4                         Ok          Presence_Detected   NA          NA          
DIMM A5                         Ok          Presence_Detected   NA          NA          
DIMM A6                         Ok          Presence_Detected   NA          NA          
DIMM A7                         N/A         Absent              NA          NA          
DIMM A8                         N/A         Absent              NA          NA          
DIMM A9                         N/A         Absent              NA          NA          
DIMM A10                        N/A         Absent              NA          NA          
DIMM B1                         Ok          Presence_Detected   NA          NA          
DIMM B2                         Ok          Presence_Detected   NA          NA          
DIMM B3                         Ok          Presence_Detected   NA          NA          
DIMM B4                         Ok          Presence_Detected   NA          NA          
DIMM B5                         Ok          Presence_Detected   NA          NA          
DIMM B6                         Ok          Presence_Detected   NA          NA          

Sensor Type : BATTERY
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board CMOS Battery       Ok          Present             NA          NA          

Sensor Type : PERFORMANCE
<Sensor Name>                   <Status>    <State>        <lc>      <uc>      
System Board Power Optimized    Ok          Not Degraded   NA        NA        

Sensor Type : INTRUSION
<Sensor Name>                   <Intrusion>    <Status>       
System Board Intrusion          Closed         Power ON       

Sensor Type : REDUNDANCY
<Sensor Name>                   <Status>                 <Type>         
System Board PS Redundancy      Full Redundant           PSU            
System Board Fan Redundancy     Full Redundant           Fan            

Sensor Type : SYSTEM PERFORMANCE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#SystemBoardIOUsage]
System Board IO Usage    Ok             0%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardMEMUsage]
System Board MEM Usage   Ok             5%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardSYSUsage]
System Board SYS Usage   Ok             10%     NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardCPUUsage]
System Board CPU Usage   Ok             13%     NA  NA    NA  [N]      101% [Y]

/admin1-> exit
CLP Session terminated
Connection to cp1076.mgmt.eqiad.wmnet closed.
Killed by signal 1.

robh@gir:~
10 $> ssh root@cp1081.mgmt.eqiad.wmnet
root@cp1081.mgmt.eqiad.wmnet's password: 
/admin1-> racadm getsensorinfo

Sensor Type : POWER
<Sensor Name>                   <Status>                 <Type>         
PS1 Status                      Present                  AC             
PS2 Status                      Present                  AC             

Sensor Type : TEMPERATURE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#CPU1Temp]
CPU1 Temp                     Ok      73C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#CPU2Temp]
CPU2 Temp                     Ok      77C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#SystemBoardInletTemp]
System Board Inlet Temp       Ok      22C      -7C  47C      3C [Y]      43C [Y]

[Key = iDRAC.Embedded.1#SystemBoardExhaustTemp]
System Board Exhaust Temp     Ok      55C      3C   80C      8C [N]      75C [N]



Sensor Type : FAN
<Sensor Name>                   <Status>    <Reading>   <lc>        <uc>        <PWM %>     
System Board Fan1A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan1B              Ok          3720RPM     480RPM      NA          14%         
System Board Fan2A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan2B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan3A              Ok          3720RPM     480RPM      NA          14%         
System Board Fan3B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan4A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan4B              Ok          4080RPM     480RPM      NA          14%         
System Board Fan5A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan5B              Ok          4080RPM     480RPM      NA          14%         
System Board Fan6A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan6B              Ok          3840RPM     480RPM      NA          14%         

Sensor Type : VOLTAGE
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board DIMM PG            Ok          Good                NA          NA          
System Board PS1 PG FAIL        Ok          Good                NA          NA          
System Board PS2 PG FAIL        Ok          Good                NA          NA          
System Board BP1 PG             Ok          Good                NA          NA          
System Board 1.8V SW PG         Ok          Good                NA          NA          
System Board 2.5V SW PG         Ok          Good                NA          NA          
System Board 5V SW PG           Ok          Good                NA          NA          
System Board PVNN SW PG         Ok          Good                NA          NA          
System Board VSB11 SW PG        Ok          Good                NA          NA          
System Board VSBM SW PG         Ok          Good                NA          NA          
System Board 3.3V B PG          Ok          Good                NA          NA          
CPU1 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM012 VPP PG              Ok          Good                NA          NA          
CPU1 MEM012 VTT PG              Ok          Good                NA          NA          
CPU1 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM345 VPP PG              Ok          Good                NA          NA          
CPU1 MEM345 VTT PG              Ok          Good                NA          NA          
CPU1 VCCIO PG                   Ok          Good                NA          NA          
CPU1 VCORE PG                   Ok          Good                NA          NA          
CPU1 FIVR PG                    Ok          Good                NA          NA          
CPU2 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM012 VPP PG              Ok          Good                NA          NA          
CPU2 MEM012 VTT PG              Ok          Good                NA          NA          
CPU2 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM345 VPP PG              Ok          Good                NA          NA          
CPU2 MEM345 VTT PG              Ok          Good                NA          NA          
CPU2 VCCIO PG                   Ok          Good                NA          NA          
CPU2 VCORE PG                   Ok          Good                NA          NA          
CPU2 FIVR PG                    Ok          Good                NA          NA          
PS1 Voltage 1                   Ok          208.00V             NA          NA          
PS2 Voltage 2                   Ok          208.00V             NA          NA          
CPU1 VSA PG                     Ok          Good                NA          NA          
CPU2 VSA PG                     Ok          Good                NA          NA          

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      220Watts NA   638Watts 0Watts [N]      594Watts [Y]


Sensor Type : PROCESSOR
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
CPU1 Status                     Ok          Presence_Detected   NA          NA          
CPU2 Status                     Ok          Presence_Detected   NA          NA          

Sensor Type : MEMORY
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
DIMM A1                         Ok          Presence_Detected   NA          NA          
DIMM A2                         Ok          Presence_Detected   NA          NA          
DIMM A3                         Ok          Presence_Detected   NA          NA          
DIMM A4                         Ok          Presence_Detected   NA          NA          
DIMM A5                         Ok          Presence_Detected   NA          NA          
DIMM A6                         Ok          Presence_Detected   NA          NA          
DIMM A7                         N/A         Absent              NA          NA          
DIMM A8                         N/A         Absent              NA          NA          
DIMM A9                         N/A         Absent              NA          NA          
DIMM A10                        N/A         Absent              NA          NA          
DIMM B1                         Ok          Presence_Detected   NA          NA          
DIMM B2                         Ok          Presence_Detected   NA          NA          
DIMM B3                         Ok          Presence_Detected   NA          NA          
DIMM B4                         Ok          Presence_Detected   NA          NA          
DIMM B5                         Ok          Presence_Detected   NA          NA          
DIMM B6                         Ok          Presence_Detected   NA          NA          

Sensor Type : BATTERY
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board CMOS Battery       Ok          Present             NA          NA          

Sensor Type : PERFORMANCE
<Sensor Name>                   <Status>    <State>        <lc>      <uc>      
System Board Power Optimized    Ok          Not Degraded   NA        NA        

Sensor Type : INTRUSION
<Sensor Name>                   <Intrusion>    <Status>       
System Board Intrusion          Closed         Power ON       

Sensor Type : REDUNDANCY
<Sensor Name>                   <Status>                 <Type>         
System Board PS Redundancy      Full Redundant           PSU            
System Board Fan Redundancy     Full Redundant           Fan            

Sensor Type : SYSTEM PERFORMANCE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#SystemBoardIOUsage]
System Board IO Usage    Ok             0%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardMEMUsage]
System Board MEM Usage   Ok             10%     NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardSYSUsage]
System Board SYS Usage   Ok             11%     NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardCPUUsage]
System Board CPU Usage   Ok             0%      NA  NA    NA  [N]      101% [Y]

/admin1-> exit
CLP Session terminated
Connection to cp1081.mgmt.eqiad.wmnet closed.
Killed by signal 1.

robh@gir:~
11 $> ssh root@cp1082.mgmt.eqiad.wmnet
root@cp1082.mgmt.eqiad.wmnet's password: 
/admin1-> racadm getsensorinfo

Sensor Type : POWER
<Sensor Name>                   <Status>                 <Type>         
PS1 Status                      Present                  AC             
PS2 Status                      Present                  AC             

Sensor Type : TEMPERATURE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#CPU1Temp]
CPU1 Temp                     Ok      73C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#CPU2Temp]
CPU2 Temp                     Ok      65C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#SystemBoardInletTemp]
System Board Inlet Temp       Ok      22C      -7C  47C      3C [Y]      43C [Y]

[Key = iDRAC.Embedded.1#SystemBoardExhaustTemp]
System Board Exhaust Temp     Ok      53C      3C   80C      8C [N]      75C [N]



Sensor Type : FAN
<Sensor Name>                   <Status>    <Reading>   <lc>        <uc>        <PWM %>     
System Board Fan1A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan1B              Ok          3600RPM     480RPM      NA          14%         
System Board Fan2A              Ok          3480RPM     480RPM      NA          14%         
System Board Fan2B              Ok          3720RPM     480RPM      NA          14%         
System Board Fan3A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan3B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan4A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan4B              Ok          3960RPM     480RPM      NA          14%         
System Board Fan5A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan5B              Ok          3960RPM     480RPM      NA          14%         
System Board Fan6A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan6B              Ok          3840RPM     480RPM      NA          14%         

Sensor Type : VOLTAGE
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board DIMM PG            Ok          Good                NA          NA          
System Board PS1 PG FAIL        Ok          Good                NA          NA          
System Board PS2 PG FAIL        Ok          Good                NA          NA          
System Board BP1 PG             Ok          Good                NA          NA          
System Board 1.8V SW PG         Ok          Good                NA          NA          
System Board 2.5V SW PG         Ok          Good                NA          NA          
System Board 5V SW PG           Ok          Good                NA          NA          
System Board PVNN SW PG         Ok          Good                NA          NA          
System Board VSB11 SW PG        Ok          Good                NA          NA          
System Board VSBM SW PG         Ok          Good                NA          NA          
System Board 3.3V B PG          Ok          Good                NA          NA          
CPU1 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM012 VPP PG              Ok          Good                NA          NA          
CPU1 MEM012 VTT PG              Ok          Good                NA          NA          
CPU1 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM345 VPP PG              Ok          Good                NA          NA          
CPU1 MEM345 VTT PG              Ok          Good                NA          NA          
CPU1 VCCIO PG                   Ok          Good                NA          NA          
CPU1 VCORE PG                   Ok          Good                NA          NA          
CPU1 FIVR PG                    Ok          Good                NA          NA          
CPU2 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM012 VPP PG              Ok          Good                NA          NA          
CPU2 MEM012 VTT PG              Ok          Good                NA          NA          
CPU2 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM345 VPP PG              Ok          Good                NA          NA          
CPU2 MEM345 VTT PG              Ok          Good                NA          NA          
CPU2 VCCIO PG                   Ok          Good                NA          NA          
CPU2 VCORE PG                   Ok          Good                NA          NA          
CPU2 FIVR PG                    Ok          Good                NA          NA          
PS2 Voltage 2                   Ok          208.00V             NA          NA          
CPU1 VSA PG                     Ok          Good                NA          NA          
CPU2 VSA PG                     Ok          Good                NA          NA          
PS1 Voltage 1                   Ok          208.00V             NA          NA          

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      198Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]


Sensor Type : PROCESSOR
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
CPU1 Status                     Ok          Presence_Detected   NA          NA          
CPU2 Status                     Ok          Presence_Detected   NA          NA          

Sensor Type : MEMORY
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
DIMM A1                         Ok          Presence_Detected   NA          NA          
DIMM A2                         Ok          Presence_Detected   NA          NA          
DIMM A3                         Ok          Presence_Detected   NA          NA          
DIMM A4                         Ok          Presence_Detected   NA          NA          
DIMM A5                         Ok          Presence_Detected   NA          NA          
DIMM A6                         Ok          Presence_Detected   NA          NA          
DIMM A7                         N/A         Absent              NA          NA          
DIMM A8                         N/A         Absent              NA          NA          
DIMM A9                         N/A         Absent              NA          NA          
DIMM A10                        N/A         Absent              NA          NA          
DIMM B1                         Ok          Presence_Detected   NA          NA          
DIMM B2                         Ok          Presence_Detected   NA          NA          
DIMM B3                         Ok          Presence_Detected   NA          NA          
DIMM B4                         Ok          Presence_Detected   NA          NA          
DIMM B5                         Ok          Presence_Detected   NA          NA          
DIMM B6                         Ok          Presence_Detected   NA          NA          

Sensor Type : BATTERY
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board CMOS Battery       Ok          Present             NA          NA          

Sensor Type : PERFORMANCE
<Sensor Name>                   <Status>    <State>        <lc>      <uc>      
System Board Power Optimized    Ok          Not Degraded   NA        NA        

Sensor Type : INTRUSION
<Sensor Name>                   <Intrusion>    <Status>       
System Board Intrusion          Closed         Power ON       

Sensor Type : REDUNDANCY
<Sensor Name>                   <Status>                 <Type>         
System Board PS Redundancy      Full Redundant           PSU            
System Board Fan Redundancy     Full Redundant           Fan            

Sensor Type : SYSTEM PERFORMANCE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#SystemBoardIOUsage]
System Board IO Usage    Ok             0%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardMEMUsage]
System Board MEM Usage   Ok             5%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardSYSUsage]
System Board SYS Usage   Ok             9%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardCPUUsage]
System Board CPU Usage   Ok             12%     NA  NA    NA  [N]      101% [Y]

/admin1-> exit
CLP Session terminated
Connection to cp1082.mgmt.eqiad.wmnet closed.
Killed by signal 1.

robh@gir:~
12 $> ssh root@cp1087.mgmt.eqiad.wmnet
root@cp1087.mgmt.eqiad.wmnet's password: 
/admin1-> racadm getsensorinfo

Sensor Type : POWER
<Sensor Name>                   <Status>                 <Type>         
PS1 Status                      Present                  AC             
PS2 Status                      Present                  AC             

Sensor Type : TEMPERATURE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#CPU1Temp]
CPU1 Temp                     Ok      75C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#CPU2Temp]
CPU2 Temp                     Ok      69C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#SystemBoardInletTemp]
System Board Inlet Temp       Ok      22C      -7C  47C      3C [Y]      43C [Y]

[Key = iDRAC.Embedded.1#SystemBoardExhaustTemp]
System Board Exhaust Temp     Ok      53C      3C   80C      8C [N]      75C [N]



Sensor Type : FAN
<Sensor Name>                   <Status>    <Reading>   <lc>        <uc>        <PWM %>     
System Board Fan1A              Ok          3480RPM     480RPM      NA          15%         
System Board Fan1B              Ok          3720RPM     480RPM      NA          15%         
System Board Fan2A              Ok          3600RPM     480RPM      NA          15%         
System Board Fan2B              Ok          3840RPM     480RPM      NA          15%         
System Board Fan3A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan3B              Ok          3960RPM     480RPM      NA          14%         
System Board Fan4A              Ok          3720RPM     480RPM      NA          14%         
System Board Fan4B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan5A              Ok          3600RPM     480RPM      NA          14%         
System Board Fan5B              Ok          3840RPM     480RPM      NA          14%         
System Board Fan6A              Ok          3720RPM     480RPM      NA          14%         
System Board Fan6B              Ok          3960RPM     480RPM      NA          14%         

Sensor Type : VOLTAGE
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board DIMM PG            Ok          Good                NA          NA          
System Board PS1 PG FAIL        Ok          Good                NA          NA          
System Board PS2 PG FAIL        Ok          Good                NA          NA          
System Board BP1 PG             Ok          Good                NA          NA          
System Board 1.8V SW PG         Ok          Good                NA          NA          
System Board 2.5V SW PG         Ok          Good                NA          NA          
System Board 5V SW PG           Ok          Good                NA          NA          
System Board PVNN SW PG         Ok          Good                NA          NA          
System Board VSB11 SW PG        Ok          Good                NA          NA          
System Board VSBM SW PG         Ok          Good                NA          NA          
System Board 3.3V B PG          Ok          Good                NA          NA          
CPU1 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM012 VPP PG              Ok          Good                NA          NA          
CPU1 MEM012 VTT PG              Ok          Good                NA          NA          
CPU1 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM345 VPP PG              Ok          Good                NA          NA          
CPU1 MEM345 VTT PG              Ok          Good                NA          NA          
CPU1 VCCIO PG                   Ok          Good                NA          NA          
CPU1 VCORE PG                   Ok          Good                NA          NA          
CPU1 FIVR PG                    Ok          Good                NA          NA          
CPU2 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM012 VPP PG              Ok          Good                NA          NA          
CPU2 MEM012 VTT PG              Ok          Good                NA          NA          
CPU2 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM345 VPP PG              Ok          Good                NA          NA          
CPU2 MEM345 VTT PG              Ok          Good                NA          NA          
CPU2 VCCIO PG                   Ok          Good                NA          NA          
CPU2 VCORE PG                   Ok          Good                NA          NA          
CPU2 FIVR PG                    Ok          Good                NA          NA          
PS1 Voltage 1                   Ok          210.00V             NA          NA          
CPU1 VSA PG                     Ok          Good                NA          NA          
CPU2 VSA PG                     Ok          Good                NA          NA          
PS2 Voltage 2                   Ok          210.00V             NA          NA          

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.6Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      198Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]


Sensor Type : PROCESSOR
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
CPU1 Status                     Ok          Presence_Detected   NA          NA          
CPU2 Status                     Ok          Presence_Detected   NA          NA          

Sensor Type : MEMORY
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
DIMM A1                         Ok          Presence_Detected   NA          NA          
DIMM A2                         Ok          Presence_Detected   NA          NA          
DIMM A3                         Ok          Presence_Detected   NA          NA          
DIMM A4                         Ok          Presence_Detected   NA          NA          
DIMM A5                         Ok          Presence_Detected   NA          NA          
DIMM A6                         Ok          Presence_Detected   NA          NA          
DIMM A7                         N/A         Absent              NA          NA          
DIMM A8                         N/A         Absent              NA          NA          
DIMM A9                         N/A         Absent              NA          NA          
DIMM A10                        N/A         Absent              NA          NA          
DIMM B1                         Ok          Presence_Detected   NA          NA          
DIMM B2                         Ok          Presence_Detected   NA          NA          
DIMM B3                         Ok          Presence_Detected   NA          NA          
DIMM B4                         Ok          Presence_Detected   NA          NA          
DIMM B5                         Ok          Presence_Detected   NA          NA          
DIMM B6                         Ok          Presence_Detected   NA          NA          

Sensor Type : BATTERY
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board CMOS Battery       Ok          Present             NA          NA          

Sensor Type : PERFORMANCE
<Sensor Name>                   <Status>    <State>        <lc>      <uc>      
System Board Power Optimized    Ok          Not Degraded   NA        NA        

Sensor Type : INTRUSION
<Sensor Name>                   <Intrusion>    <Status>       
System Board Intrusion          Closed         Power ON       

Sensor Type : REDUNDANCY
<Sensor Name>                   <Status>                 <Type>         
System Board PS Redundancy      Full Redundant           PSU            
System Board Fan Redundancy     Full Redundant           Fan            

Sensor Type : SYSTEM PERFORMANCE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#SystemBoardIOUsage]
System Board IO Usage    Ok             0%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardMEMUsage]
System Board MEM Usage   Ok             9%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardSYSUsage]
System Board SYS Usage   Ok             21%     NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardCPUUsage]
System Board CPU Usage   Ok             26%     NA  NA    NA  [N]      101% [Y]

/admin1-> exit
CLP Session terminated
Connection to cp1087.mgmt.eqiad.wmnet closed.
Killed by signal 1.

robh@gir:~
13 $> ssh root@cp1088.mgmt.eqiad.wmnet
root@cp1088.mgmt.eqiad.wmnet's password: 
/admin1-> racadm getsensorinfo

Sensor Type : POWER
<Sensor Name>                   <Status>                 <Type>         
PS1 Status                      Present                  AC             
PS2 Status                      Present                  AC             

Sensor Type : TEMPERATURE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#CPU1Temp]
CPU1 Temp                     Ok      66C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#CPU2Temp]
CPU2 Temp                     Ok      63C      3C   93C      NA [N]      NA [N]

[Key = iDRAC.Embedded.1#SystemBoardInletTemp]
System Board Inlet Temp       Ok      23C      -7C  47C      3C [Y]      43C [Y]

[Key = iDRAC.Embedded.1#SystemBoardExhaustTemp]
System Board Exhaust Temp     Ok      50C      3C   80C      8C [N]      75C [N]



Sensor Type : FAN
<Sensor Name>                   <Status>    <Reading>   <lc>        <uc>        <PWM %>     
System Board Fan1A              Ok          3600RPM     480RPM      NA          15%         
System Board Fan1B              Ok          3840RPM     480RPM      NA          15%         
System Board Fan2A              Ok          3720RPM     480RPM      NA          15%         
System Board Fan2B              Ok          3960RPM     480RPM      NA          15%         
System Board Fan3A              Ok          3840RPM     480RPM      NA          15%         
System Board Fan3B              Ok          4560RPM     480RPM      NA          15%         
System Board Fan4A              Ok          4200RPM     480RPM      NA          15%         
System Board Fan4B              Ok          4080RPM     480RPM      NA          15%         
System Board Fan5A              Ok          4320RPM     480RPM      NA          15%         
System Board Fan5B              Ok          3960RPM     480RPM      NA          15%         
System Board Fan6A              Ok          4080RPM     480RPM      NA          15%         
System Board Fan6B              Ok          3840RPM     480RPM      NA          15%         

Sensor Type : VOLTAGE
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board DIMM PG            Ok          Good                NA          NA          
System Board PS1 PG FAIL        Ok          Good                NA          NA          
System Board PS2 PG FAIL        Ok          Good                NA          NA          
System Board BP1 PG             Ok          Good                NA          NA          
System Board 1.8V SW PG         Ok          Good                NA          NA          
System Board 2.5V SW PG         Ok          Good                NA          NA          
System Board 5V SW PG           Ok          Good                NA          NA          
System Board PVNN SW PG         Ok          Good                NA          NA          
System Board VSB11 SW PG        Ok          Good                NA          NA          
System Board VSBM SW PG         Ok          Good                NA          NA          
System Board 3.3V B PG          Ok          Good                NA          NA          
CPU1 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM012 VPP PG              Ok          Good                NA          NA          
CPU1 MEM012 VTT PG              Ok          Good                NA          NA          
CPU1 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU1 MEM345 VPP PG              Ok          Good                NA          NA          
CPU1 MEM345 VTT PG              Ok          Good                NA          NA          
CPU1 VCCIO PG                   Ok          Good                NA          NA          
CPU1 VCORE PG                   Ok          Good                NA          NA          
CPU1 FIVR PG                    Ok          Good                NA          NA          
CPU2 MEM012 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM012 VPP PG              Ok          Good                NA          NA          
CPU2 MEM012 VTT PG              Ok          Good                NA          NA          
CPU2 MEM345 VDDQ PG             Ok          Good                NA          NA          
CPU2 MEM345 VPP PG              Ok          Good                NA          NA          
CPU2 MEM345 VTT PG              Ok          Good                NA          NA          
CPU2 VCCIO PG                   Ok          Good                NA          NA          
CPU2 VCORE PG                   Ok          Good                NA          NA          
CPU2 FIVR PG                    Ok          Good                NA          NA          
PS2 Voltage 2                   Ok          210.00V             NA          NA          
CPU1 VSA PG                     Ok          Good                NA          NA          
CPU2 VSA PG                     Ok          Good                NA          NA          
PS1 Voltage 1                   Ok          210.00V             NA          NA          

Sensor Type : CURRENT
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#PS2Current2]
PS2 Current 2                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]

[Key = iDRAC.Embedded.1#SystemBoardPwrConsumption]
System Board Pwr Consumption  Ok      176Watts NA   638Watts 0Watts [N]      594Watts [Y]

[Key = iDRAC.Embedded.1#PS1Current1]
PS1 Current 1                 Ok      0.4Amps  NA   NA       0Amps [N]      0Amps [N]


Sensor Type : PROCESSOR
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
CPU1 Status                     Ok          Presence_Detected   NA          NA          
CPU2 Status                     Ok          Presence_Detected   NA          NA          

Sensor Type : MEMORY
<Sensor Name>                   <Status>    <State>             <lc>        <uc>        
DIMM A1                         Ok          Presence_Detected   NA          NA          
DIMM A2                         Ok          Presence_Detected   NA          NA          
DIMM A3                         Ok          Presence_Detected   NA          NA          
DIMM A4                         Ok          Presence_Detected   NA          NA          
DIMM A5                         Ok          Presence_Detected   NA          NA          
DIMM A6                         Ok          Presence_Detected   NA          NA          
DIMM A7                         N/A         Absent              NA          NA          
DIMM A8                         N/A         Absent              NA          NA          
DIMM A9                         N/A         Absent              NA          NA          
DIMM A10                        N/A         Absent              NA          NA          
DIMM B1                         Ok          Presence_Detected   NA          NA          
DIMM B2                         Ok          Presence_Detected   NA          NA          
DIMM B3                         Ok          Presence_Detected   NA          NA          
DIMM B4                         Ok          Presence_Detected   NA          NA          
DIMM B5                         Ok          Presence_Detected   NA          NA          
DIMM B6                         Ok          Presence_Detected   NA          NA          

Sensor Type : BATTERY
<Sensor Name>                   <Status>    <Reading>           <lc>        <uc>        
System Board CMOS Battery       Ok          Present             NA          NA          

Sensor Type : PERFORMANCE
<Sensor Name>                   <Status>    <State>        <lc>      <uc>      
System Board Power Optimized    Ok          Not Degraded   NA        NA        

Sensor Type : INTRUSION
<Sensor Name>                   <Intrusion>    <Status>       
System Board Intrusion          Closed         Power ON       

Sensor Type : REDUNDANCY
<Sensor Name>                   <Status>                 <Type>         
System Board PS Redundancy      Full Redundant           PSU            
System Board Fan Redundancy     Full Redundant           Fan            

Sensor Type : SYSTEM PERFORMANCE
<Sensor Name>            <Status>    <Reading> <lc> <uc>  <lnc>[R/W]  <unc>[R/W]
[Key = iDRAC.Embedded.1#SystemBoardIOUsage]
System Board IO Usage    Ok             0%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardMEMUsage]
System Board MEM Usage   Ok             4%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardSYSUsage]
System Board SYS Usage   Ok             7%      NA  NA    NA  [N]      101% [Y]

[Key = iDRAC.Embedded.1#SystemBoardCPUUsage]
System Board CPU Usage   Ok             9%      NA  NA    NA  [N]      101% [Y]

/admin1->

I'll parse out the power details from the above and update the task summary on Friday AM.

RobH updated the task description. (Show Details)Jun 25 2019, 5:56 PM
RobH added a comment.EditedJun 25 2019, 6:02 PM

Ok, using the data I've summarized from the above outputs, we have the following rough power draws at peak hours:

power figures for 2019-07-20

@RobH pulled the data in comments above this one, summarized here:

hostnamedevice typewattage
cr1-eqiadMX4801488 W
cr2-eqiadMX4801425 W
asw2-b2-eqiadQFX510048 W
asw2-b4-eqiadQFX510048 W
asw2-b7-eqiadQFX510048 W
asw2-d2-eqiadQFX510048 W
asw2-d4-eqiadQFX510048 W
asw2-d7-eqiadQFX510048 W
cp1075R440 (cp text)198 W
cp1081R440 (cp text)220 W
cp1087R440 (cp text)198 W
cp1076R440 (cp upload)198 W
cp1082R440 (cp upload)198 W
cp1088R440 (cp upload)176 W

So, we should plan for each R440 to require 250 W as its above average requirement for planning.

devicewatts# of devices required
MX4801500 W2
QFX510050 W1 per rack
R440250 W16 caching + 3 misc + 3 lvs + 3 ganeti = 25 hosts

If we go with two racks, we can have the following per rack (evenly dividing the above)
(router) 1500W + (switch) 50 W + (13 hosts) 3250 W = 4800 W per rack.

RobH added a comment.Jun 25 2019, 8:23 PM

Summarizing IRC discussion between @BBlack and @RobH:

The R440 CP systems will pull 250W each in our estimates (pulled from live data at peak in eqiad)
The R440 lvs/misc/ganeti will pull 150W in our estimates (estimated from R430 power usage for these roles)
We'll have: 16x250 + 9x150 -> 5350 total watts for the server nodes, nominal at peak-ish load times

Two 1500W routers + 5350W for all servers = 8350W, * 1.25 to keep under your 80% figure -> 10437

3000W + (5350 * 2) -> 13700 ... * 1.25 for your headroom is 17125W. Divide over 3 racks and you'd need 5.7kW ceiling to be able to bring a new set of servers online and loaded alongside the old.
Summary: 3 racks at 5.7kW ceiling for use. This is a estimate and could scale up or down to meet vendor offerings.

BBlack closed this task as Resolved.Oct 29 2019, 1:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL