Page MenuHomePhabricator

Maintain symlinks for WMCS NFS Dumps users with new directory structure
Closed, ResolvedPublic

Description

Currently, the rsync exports to labstore1003 sends data in a structure that on WMCS nfs dumps clients ends up looking like:

root@newdumps-test:/public/dumps# ls
incr  pagecounts-all-sites	pagecounts-raw	pageviews  public

In the new setup, we won't have the rsyncs anymore, and the directory structure has incr, pagecounts-all-sites, pagecounts-raw, pageviews under /other. In order to not break client code when we migrate, I'm proposing we maintain these four as symlinks for backwards compatibility with the current directory structure.

Details

Related Gerrit Patches:

Event Timeline

madhuvishy triaged this task as Medium priority.Feb 23 2018, 7:40 AM
madhuvishy created this task.

This seems fine to me provided it can be handled on the labstore boxes without requiring any changes on the dumpsdata servers. Also we might think about what a 'good' directory structure would look like, and how we could phase out such symlinks over time.

+1 to handling on labstore boxes. Puppet should be able to do it.

As for 'good' directory structures I do wonder if having a similar organization as the dumps public site would make sense - https://dumps.wikimedia.org/.

Phasing out old directory structures usually takes weeks of constant communication to Tool and VPS project owners, and we've usually seen that many folks will not respond or show up to fix their tools. This is sometimes unavoidable in cases like moving to new labsdb boxes, but even in that case we are still maintaining backward compatible dns aliases. I'm for communicating whatever new directory structures we come up with and having people move over. But I'm not optimistic about deprecating backward compatibility layers.

Of course, we could try to grep for uses of the old directory structure in source code and narrow down what and how many tools will be affected and make decisions based on that, probably not right now but if/when we make plans to move to a more cohesive directory structure.

bd808 moved this task from Backlog to Dumps on the Data-Services board.Mar 2 2018, 1:43 PM

Change 422867 had a related patch set uploaded (by Madhuvishy; owner: Madhuvishy):
[operations/puppet@production] dumps: Set up symlinks on instances under /public/dumps

https://gerrit.wikimedia.org/r/422867

madhuvishy closed this task as Resolved.Mar 29 2018, 8:26 PM

Change 422867 merged by Madhuvishy:
[operations/puppet@production] dumps: Set up symlinks on instances under /public/dumps

https://gerrit.wikimedia.org/r/422867

bd808 moved this task from Inbox to Done on the cloud-services-team (Kanban) board.May 6 2018, 6:48 PM