Page MenuHomePhabricator
Create Task
Maniphest T238032

cp3065 crashed
Closed, ResolvedPublic
Actions

Description

cp3065 went down yesterday 2019-11-11 at 21:54:24 showing the same symptoms as described in T237348 for cp3057.

The server has been power-cycled by @ema and is currently reachable but depooled.

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT238305 Servers freezing across the caching cluster
 ResolvedVgutierrezT238032 cp3065 crashed

Event Timeline

Vgutierrez created this task.Nov 12 2019, 2:29 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 12 2019, 2:29 AM
• ema triaged this task as Medium priority.Nov 12 2019, 10:05 AM
Stashbot added a comment.Nov 12 2019, 10:06 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-12T10:06:59Z] <ema> repool cp3065, nothing interesting in kern.log and SEL T238032

• ema moved this task from Backlog to Hardware on the Traffic board.Nov 12 2019, 2:45 PM
• ema added a comment.Nov 12 2019, 3:28 PM

Perhaps interestingly, or maybe entirely unrelated: a couple of hours before crashing the host had a spike in cache write errors:

Screenshot from 2019-11-12 16-26-24.png (1×2 px, 294 KB)

Vgutierrez mentioned this in T238289: cp1077 is unreachable.Nov 14 2019, 4:07 AM
Vgutierrez mentioned this in T238305: Servers freezing across the caching cluster.Nov 14 2019, 9:36 AM
Vgutierrez added a parent task: T238305: Servers freezing across the caching cluster.
Vgutierrez closed this task as Resolved.Nov 14 2019, 11:32 AM
Vgutierrez claimed this task.

Tracking the issue on the parent task: T238305

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits