Page MenuHomePhabricator
Create Task
Maniphest T238032

cp3065 crashed
Closed, ResolvedPublic
Actions

Description

cp3065 went down yesterday 2019-11-11 at 21:54:24 showing the same symptoms as described in T237348 for cp3057.

The server has been power-cycled by @ema and is currently reachable but depooled.

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT238305 Servers freezing across the caching cluster
 ResolvedVgutierrezT238032 cp3065 crashed

Event Timeline

Vgutierrez created this task.Nov 12 2019, 2:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 12 2019, 2:29 AM
• ema triaged this task as Medium priority.Nov 12 2019, 10:05 AM
Stashbot added a comment.Nov 12 2019, 10:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T10:06:59Z] <ema> repool cp3065, nothing interesting in kern.log and SEL T238032

• ema moved this task from Triage to Hardware on the Traffic board.Nov 12 2019, 2:45 PM
• ema added a comment.Nov 12 2019, 3:28 PM

Perhaps interestingly, or maybe entirely unrelated: a couple of hours before crashing the host had a spike in cache write errors:

Screenshot from 2019-11-12 16-26-24.png (1×2 px, 294 KB)

Vgutierrez mentioned this in T238289: cp1077 is unreachable.Nov 14 2019, 4:07 AM
Vgutierrez mentioned this in T238305: Servers freezing across the caching cluster.Nov 14 2019, 9:36 AM
Vgutierrez added a parent task: T238305: Servers freezing across the caching cluster.
Vgutierrez closed this task as Resolved.Nov 14 2019, 11:32 AM
Vgutierrez claimed this task.

Tracking the issue on the parent task: T238305

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL