output error
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Dzahn
	Apr 19 2019, 10:44 PM

Description

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=furud&service=Disk+space

host furud: DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error

# Backup system, see T176506.
# This is a reserved system. Ask Otto or Faidon.
node 'furud.codfw.wmnet' {
    role(analytics_cluster::hadoop::client)

16:07 <+icinga-wm> PROBLEM - Disk space on furud is CRITICAL: DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space

16:09 < mutante> ottomata: furud ^ seems out of disk. site.pp says it's a hadoop client and backup and to ask you

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Dzahn	T221483 furud - DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error
		Declined		Ottomata	T221629 Remove Hadoop configs and unmount /mnt/hdfs from unused backup hosts (furud, +)

Event Timeline

Dzahn created this task.Apr 19 2019, 10:44 PM

Restricted Application added a project: Analytics. · View Herald TranscriptApr 19 2019, 10:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-operations) [2019-04-19T22:47:11Z] <mutante> furud - remounted /mnt/hdfs for T221483

followed the docs at https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_/mnt/hdfs

but had to repeat commands twice:

[furud:~] $  sudo umount -f /mnt/hdfs
[furud:~] $  sudo fusermount -uz /mnt/hdfs
fusermount: failed to unmount /mnt/hdfs: Invalid argument
[furud:~] $  sudo mount /mnt/hdfs
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:164 Adding FUSE arg /mnt/hdfs
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:115 Ignoring option allow_other
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:115 Ignoring option dev
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:115 Ignoring option suid
[furud:~] $  sudo fusermount -uz /mnt/hdfs
[furud:~] $  sudo mount /mnt/hdfs
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:164 Adding FUSE arg /mnt/hdfs
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:115 Ignoring option allow_other
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:115 Ignoring option dev
INFO /data/jenkins/workspace/generic-package-debian64-8-0/CDH5.16.1-Packaging-Hadoop-2018-11-21_20-46-52/hadoop-2.6.0+cdh5.16.1+2848-1.cdh5.16.1.p0.3~jessie/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:115 Ignoring option suid
[furud:~] $

18:46 <+icinga-wm> RECOVERY - Disk space on furud is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space

Many thanks @Dzahn

Peachey88 updated the task description. (Show Details)Apr 20 2019, 12:43 AM

Thanks! We should probably unpuppetize the Hadoop part of these nodes and unmount /mnt/hdfs until we need them again.

Ottomata reassigned this task from Ottomata to Dzahn.Apr 23 2019, 2:16 PM

Ottomata closed subtask T221629: Remove Hadoop configs and unmount /mnt/hdfs from unused backup hosts (furud, +) as Declined.Apr 24 2019, 2:55 PM

Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM

furud - DISK CRITICAL - /mnt/hdfs is not accessible: Input/output errorClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

furud - DISK CRITICAL - /mnt/hdfs is not accessible: Input/output error
Closed, ResolvedPublic
Actions

Related Objects
Search...