Page MenuHomePhabricator

Deprecate /mnt/hdfs from the Analytics infra
Closed, DeclinedPublic

Description

The /mnt/hdfs mount is brittle and cause of operational headaches, we should think about deprecating it.

Related Objects

Event Timeline

Heh, interesting, but it is very useful! Do the operational headaches outweigh its usefulness? I don't think they do, but you are more exposed to the headaches these days than I am!

Heh, interesting, but it is very useful! Do the operational headaches outweigh its usefulness? I don't think they do, but you are more exposed to the headaches these days than I am!

My experience as user of /mnt/hdfs is not really great as well, it happens to me a lot of times that simply doing an ls to /mnt/hdfs/something takes ages. During the past month it was not a big deal but in the past months Ariel pinged me several times asking to check /mnt/hdfs on stat1007 because rsyncs were failing (and the usual fix was umount/mount). What are the useful points of using this mount point if it is so brittle?

Hm, that isn't my experience. That does occasionally happen to me, but being able to cd into and around HDFS directories makes the experience of using HDFS much easier, especially because of tab autocompletion. hdfs dfs -ls deep into hive partition directories is pretty cumbersome, especially when you are just trying to find an example to use, not looking for a specific path.

fdans triaged this task as Medium priority.Dec 23 2019, 4:38 PM
fdans moved this task from Incoming to Operational Excellence on the Analytics board.
fdans added subscribers: JAllemandou, fdans.

@JAllemandou agrees with this 🍕

After introducing the hdfs-rsync tool, all the mount points have been really stable. I am inclined to close this task and re-open if needed.