Page MenuHomePhabricator

stat1004 - /mnt/hdfs is not accessible
Closed, ResolvedPublic

Description

DISK CRITICAL - /mnt/hdfs is not accessible: Transport endpoint is not connected

started: 1d 13h 52m 27s ago

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=stat1004&service=Disk+space

Event Timeline

Dzahn created this task.Dec 7 2017, 6:32 PM
Restricted Application added a project: Analytics. · View Herald TranscriptDec 7 2017, 6:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-operations) [2018-09-28T20:57:09Z] <mutante> analytics1003 - unmounted and remounted /mnt/hdfs after Icinga alerts that it was not accessible - commands from https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_/mnt/hdfs - like it happened before on stat1004 and others (T182342)

Dzahn added a comment.Sep 28 2018, 8:58 PM

16:45 < icinga-wm> PROBLEM - Disk space on analytics1003 is CRITICAL: DISK CRITICAL - /mnt/hdfs is not accessible: No such file or directory
16:50 < icinga-wm> PROBLEM - Disk space on analytics1003 is CRITICAL: DISK CRITICAL - /mnt/hdfs is not accessible: No such file or directory
..
16:54 < icinga-wm> RECOVERY - Disk space on analytics1003 is OK: DISK OK

Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM