Page MenuHomePhabricator

Make dumps accessible in PAWS
Closed, ResolvedPublic

Description

Task to define a method and policy to update said method of making dumps accessible in PAWS. At the moment PAWS uses the symbolic link defined at /public/dumps/incr to find what path dumps is and then mounts it to /public/dumps inside each user pod. This still feels a little hacky and prone to problems in the future, but might be the best way handle this until we are free of NFS.

Description was:
At the moment PAWS does mount /public/dumps in the users pods, but since that contains symlinks to the actual mounted NFS shares they are not accessible.

Event Timeline

One option we have is to do a simple volume mount of /mnt/nfs/dumps-labstore1006.wikimedia.org and that should make the symlink work.

Chicocvenancio claimed this task.

Done with host volume to /mnt/nfs/dumps-labstore1006.wikimedia.org. This is non ideal as any changes to that nfs mount structure will break things.

https://github.com/yuvipanda/paws/commit/4a79ac21a7f4807fec0548cdcbecb0f96e650e83

Chicocvenancio lowered the priority of this task from Medium to Low.
Chicocvenancio added a subscriber: chasemp.

As expected, the change to dumps-labstore1007.wikimedia.org broke this. I went ahead with a, slightly less brittle hack of reading the symlink from '/public/dumps/incr' to find what is the currently used nfs mount for dumps.

Still think this isn't ideal, @chasemp, do we have somewhere we can query for that info beyond the symlinks?

Just to clarify for casual readers: the dumps are currently accessible. (I used them successfully in this notebook earlier this month, thanks @Chicocvenancio for fixing this earlier!) I understand this ticket is now about implementing this in a more future-proof manner - should the task description be updated?

Just to clarify for casual readers: the dumps are currently accessible. (I used them successfully in this notebook earlier this month, thanks @Chicocvenancio for fixing this earlier!) I understand this ticket is now about implementing this in a more future-proof manner - should the task description be updated?

Done, thanks for the reminder.

Bstorm added a subscriber: Bstorm.

I've set up a pretty solid dumps mount in the new paws cluster at this point, and the symlinks should work as expected (and just like in Toolforge).