#!/bin/bash

cd /srv/backup/tools/shared/tools/project
# 566f5ecc3bf0f3bdbd90a609cf966f70  /home/rush/tools-permissive-files
wc -l `cat /home/rush/tools-permissive-files | grep -v '^#'`
for f in `cat /home/rush/tools-permissive-files | grep -v '^#'`; do
    # tool project
    project=`echo $f | cut -d '/' -f 5`
    # relative path from project name
    restore_me=`echo $f | awk -F $project '{print $2}'`

    echo $project
    echo $restore_me

    rsync \
        --timeout=30 --bwlimit=250000 \
        --archive --hard-links --inplace \
        --links --perms --times --relative -r \
        --rsh 'ssh -C -i /root/.ssh/id_labstore' \
        "${project}${restore_me}" \
        root@labstore1003.eqiad.wmnet:/srv/scratch/T169774/
done

(as root)# touch foo/bar/vuln/myfile
(as root)# tree
.
└── foo
    └── bar
        └── vuln
            └── myfile

3 directories, 1 file
(as root)# ls -ald foo/bar/vuln
drwxrwxr-x 2 root root 4096 Jul  7 19:50 foo/bar/vuln

(as root)# ls -al foo/bar/vuln/myfile
-rw-r--r-- 1 root root 0 Jul  7 19:50 foo/bar/vuln/myfile

(as root)# ls -ald foo/bar/vuln
drwxrwxr-x 2 root root 4096 Jul  7 19:50 foo/bar/vuln

(as regular user)#rm -f foo/bar/vuln/myfile
rm: cannot remove ‘foo/bar/vuln/myfile’: Permission denied

(as root)# chmod o+w foo/bar/vuln

(as regular user)#rm -f foo/bar/vuln/myfile

(as root)# ls -ald foo/bar/vuln
drwxrwxrwx 2 root root 4096 Jul  7 19:50 foo/bar/vuln

On 2017-07-02 some users experienced a loss of data in their projects.  

We estimate 126 Tools out of 1,766 saw at least one file removed.  We have reason to believe a user acting under a Tool account issued the command 'rm -fr *' at the wrong point in the directory structure.  Anyone who had files that were removable by this user was effected.  Thankfully, we have a backup from before the command was run, a minority of users have overly permissive files, and a further minority were severely impacted.  https://phabricator.wikimedia.org/T169774 was created in response to inquiries surrounding data loss.  

We do not guarantee any level of user backups for day-to-day operations, but in this case since we do have the data I have restored it to /data/scratch/T169774/ so users can retrieve what was removed.  We intend to make this restored data available until at least 2017-08-08.  A Warning: Please do not rely on NFS for backups of code or critical data.  We only have capacity to keep 2 weeks of historical backups at the moment and cannot guarantee timely retrieval or availability.  Every Tool account can use https://phabricator.wikimedia.org/diffusion/ for code hosting, and creation of the repository is handled by going to https://toolsadmin.wikimedia.org/tools/id/<mytool>.

This calamity was almost entirely caused by directories with o+w set allowing 'other' or 'everyone' write access.  Do not use permissions such as '777' or that look like 'drwxrwxrwx' as it will allow other users to remove your files.  This is especially dangerous in a shared hosting environment as this incident has shown.

A brief explanation of why this happened to users who have given write permissions to 'other' for a directory in their Tool:

Because directories are not used in the same way as regular files, the permissions work slightly (but only slightly) differently.  An attempt to list the files in a directory requires read permission for the directory, but not on the files within.  An attempt to add a file to a directory, delete a file from a directory, or to rename a file, all require write permission for the directory, but (perhaps surprisingly) not for the files within. 

-  Unix File and Directory Permissions and Modes (https://wpollock.com/AUnix1/FilePermissions.htm)

ACTION ITEMS:
- Make sure you have backups of code and data needed
- Check for removed data you want to restore on login.tools.wmflabs.org at /data/scratch/T169774/ 
- Check your Tools files and directories for o+w permission and remove if possible (chmod -R o-w <directory>).
- Ask for help on the labs-l mailing list, Phabricator, or in the #wikimedia-cloud IRC channel if you cannot figure out how to do without o+w (someone may have a different solution).