Page MenuHomePhabricator
Create Task
Maniphest T239041

cp3053 is unreachable
Closed, ResolvedPublic
Actions

Description

cp3053 went offline on Saturday 23rd at 18:31 UTC, this seems like another occurrence of T238305.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT238305 Servers freezing across the caching cluster
 Resolved• emaT239041 cp3053 is unreachable

Event Timeline

Vgutierrez created this task.Nov 25 2019, 2:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 25 2019, 2:47 AM
Stashbot added a comment.Nov 25 2019, 2:59 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-25T02:59:24Z] <vgutierrez> depooling & power-cycling cp3053 - T239041

Vgutierrez triaged this task as Medium priority.Nov 25 2019, 3:03 AM
Vgutierrez updated the task description. (Show Details)
Vgutierrez moved this task from Triage to Hardware on the Traffic board.
Vgutierrez mentioned this in T238305: Servers freezing across the caching cluster.Nov 25 2019, 3:09 AM
Vgutierrez added a parent task: T238305: Servers freezing across the caching cluster.
Stashbot added a comment.Nov 25 2019, 3:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-25T03:13:29Z] <vgutierrez> repooling cp3053 - T239041

Vgutierrez closed this task as Resolved.Nov 25 2019, 3:14 AM
Vgutierrez claimed this task.

Nothing on the logs or on SEL

• Marostegui reopened this task as Open.Dec 2 2019, 10:25 AM
• Marostegui added a subscriber: • Marostegui.

This host went down again:

And [10:23:27]  <+icinga-wm>	PROBLEM - Host cp3053 is DOWN: PING CRITICAL - Packet loss = 100%
Stashbot added a comment.Dec 2 2019, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-02T13:47:41Z] <ema> power-cycle cp3053 T239041

MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Dec 2 2019, 1:55 PM

As mentioned in last week's SRE meeting, let's upgrade the firmware to the latest revision cpn cp3053?

MoritzMuehlenhoff added a subscriber: RobH.Dec 2 2019, 2:16 PM

@RobH As you offered help in the SRE meeting last Monday, can you upgrade the firmware on cp3053?

RobH added a comment.Dec 2 2019, 3:32 PM

It appears cp3053 is online at this time. Can I issue a depool command and shut it down for the firmware update at any time or is further scheduling needed? The process will take about 5-15 minutes.

RobH added a comment.Dec 2 2019, 4:10 PM

confirmed with @ema that depool via command line and power off is fine, moving on to flashing firmware.

Stashbot added a comment.Dec 2 2019, 4:10 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-02T16:10:51Z] <robh> cp3035 depooling and rebooting for firmware update T239041

Stashbot added a comment.Dec 2 2019, 4:11 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-02T16:11:01Z] <robh> cp3053 depooling and rebooting for firmware update T239041

RobH reassigned this task from Vgutierrez to • ema.Dec 2 2019, 4:31 PM

All ilom and bios updated, irc update to @ema and handing this back to Traffic.

Stashbot added a comment.Dec 2 2019, 4:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-02T16:32:09Z] <ema> cp3053: repooling after firmware update T239041

• ema closed this task as Resolved.Dec 9 2019, 2:07 PM

The host has now been up with the new firmware with no issues for one week. Closing for now, we can re-open if needed.

• ema mentioned this in T240425: cp3055 crashed.Dec 11 2019, 9:19 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL