Page MenuHomePhabricator
Create Task
Maniphest T302154

quarry-nfs-1 went down; quarry is offline
Closed, ResolvedPublic
Actions

Description

13:07:04 <+wmcs-alerts> (InstanceDown) firing: Project quarry instance quarry-nfs-1 is down   - https://prometheus-alerts.wmcloud.org
13:26:04 <+wmcs-alerts> (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project quarry   - https://prometheus-alerts.wmcloud.org

Related Objects

Event Timeline

RhinosF1 triaged this task as Unbreak Now! priority.Feb 19 2022, 1:38 PM
RhinosF1 created this task.
Restricted Application edited projects, added cloud-services-team (Kanban); removed cloud-services-team. · View Herald TranscriptFeb 19 2022, 1:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
TheresNoTime subscribed.Feb 19 2022, 1:58 PM
GeneralNotability subscribed.Feb 19 2022, 1:59 PM
Stashbot added a comment.Feb 19 2022, 2:04 PM

Mentioned in SAL (#wikimedia-cloud) [2022-02-19T14:04:05Z] <taavi> reboot quarry-nfs-1 T302154

taavi lowered the priority of this task from Unbreak Now! to High.Feb 19 2022, 2:11 PM
taavi subscribed.

Rebooting the NFS server seems to have solved the immediate issue.

Leaving this task open so that we can investigate why this happened and how to prevent it. The VM was fully unresponsible, even the serial console didn't react at all. The full console log is on P21049.

RhinosF1 raised the priority of this task from High to Unbreak Now!.Feb 20 2022, 7:17 PM

Crashed again

RhinosF1 merged a task: T302180: Quarry has become inaccessible ("502 Bad Gateway").Feb 20 2022, 7:18 PM
RhinosF1 added a subscriber: Huji.
RhinosF1 added a comment.Feb 20 2022, 7:21 PM

For the record:

19:02:04 <wmcs-alerts> (InstanceDown) firing: Project quarry instance quarry-nfs-1 is down  - https://prometheus-alerts.wmcloud.org
19:19:04 <wmcs-alerts> (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project quarry  - https://prometheus-alerts.wmcloud.org

RhinosF1 updated the task description. (Show Details)Feb 20 2022, 7:21 PM
Stashbot added a comment.Feb 20 2022, 7:23 PM

Mentioned in SAL (#wikimedia-cloud) [2022-02-20T19:23:35Z] <taavi> hard rebooted quarry-nfs-1 again T302154

Stashbot added a comment.Feb 20 2022, 7:49 PM

Mentioned in SAL (#wikimedia-cloud) [2022-02-20T19:49:50Z] <andrewbogott> moving nfs service from quarry-nfs-1 (bullseye) to quarry-nfs-2 (buster), testing to see if T302154 is a kernal or nfs-version issue

zhuyifei1999 subscribed.Feb 20 2022, 11:35 PM

kernel bug. RCU should not stall.

zhuyifei1999 added a comment.Feb 20 2022, 11:49 PM

The VM was fully unresponsible, even the serial console didn't react at all.

Is it possible to send a sysrq? If so, can you send a sysrq-l & sysrq-w next time? Should force a backtrace and we can see which kernel bug this is.

Nikerabbit subscribed.Feb 21 2022, 1:05 PM
RhinosF1 merged a task: T302302: quarry-nfs-1 - alertname: PuppetAgentStaleLastRun project: quarry.Feb 22 2022, 3:57 PM
RhinosF1 added a subscriber: dcaro.
zhuyifei1999 lowered the priority of this task from Unbreak Now! to High.Feb 23 2022, 5:42 AM
RhinosF1 closed this task as Resolved.Aug 8 2022, 8:04 PM

Not happened since. Closing per IRC.

Framawiki added subscribers: Andrew, Framawiki.EditedMay 27 2023, 2:44 PM
In T302154#7724373, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2022-02-20T19:49:50Z] < @Andrew > moving nfs service from quarry-nfs-1 (bullseye) to quarry-nfs-2 (buster), testing to see if T302154 is a kernal or nfs-version issue

@Andrew can quarry-nfs-1 be deleted then? I don't see anything stored on it, /srv/quarry/project is empty and has no volume mounted.
All data is on quarry-nfs-2 and this one has the external volume mounted on it.
I turned off the instance to verify it's not used

Andrew added a comment.Jun 8 2023, 2:34 PM
In T302154#8884891, @Framawiki wrote:
In T302154#7724373, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2022-02-20T19:49:50Z] < @Andrew > moving nfs service from quarry-nfs-1 (bullseye) to quarry-nfs-2 (buster), testing to see if T302154 is a kernal or nfs-version issue

@Andrew can quarry-nfs-1 be deleted then? I don't see anything stored on it, /srv/quarry/project is empty and has no volume mounted.
All data is on quarry-nfs-2 and this one has the external volume mounted on it.
I turned off the instance to verify it's not used

We have a cookbook to failover NFS between an active server and a passive one, which includes moving the volume attachment. So nfs-1 is there as a safety net. I wouldn't insist that it's a /necessary/ safety net but it also seems like a harmless one.

Andrew added a comment.Jun 8 2023, 4:53 PM

Ah, sorry, I should've read back further in the task! Yes, that host can+should be deleted.

Framawiki added a comment.Jun 22 2023, 6:23 PM
In T302154#8914904, @Andrew wrote:

Ah, sorry, I should've read back further in the task! Yes, that host can+should be deleted.

thanks, deleted.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL