Page MenuHomePhabricator

maps: take back root owned files/dirs from root_squash protected nfs
Closed, ResolvedPublic

Description

So.. the ancientness of the maps servers show their ugly face once more ;)

I'm guessing /data/project NFS is run in root_squash mode, as root users can't seem to do anything there. Unfortunately however, there are multiple files in this directory that are root:root owned. This makes it they cannot be deleted, changed etc...

Anyone know how to fix this ?

Event Timeline

TheDJ created this task.Mar 12 2019, 7:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2019, 7:21 PM
TheDJ updated the task description. (Show Details)Mar 12 2019, 7:25 PM
TheDJ assigned this task to bd808.EditedMar 12 2019, 7:50 PM

www-data:www-data will probably be good. If needed, I can claim ownership of www-data via sudo -u www-data

TheDJ added a comment.Mar 12 2019, 7:51 PM

Possibly exclude /data/project/wma, as that is currently owned by dschwen, and i don't want to interfere with that.

bd808 renamed this task from take back root owned files/dirs from root_squash protected nfs to maps: take back root owned files/dirs from root_squash protected nfs.Mar 12 2019, 8:16 PM
bd808 triaged this task as High priority.
bd808 added a comment.Mar 12 2019, 8:32 PM
$ ssh labstore1003.eqiad.wmflabs
$ cd /srv/maps/project
$ ls -lh
total 20K
drwxr-xr-x  2 root     root     4.0K Mar 27  2014 chippy/
drwxr-xr-x  7 root     root     4.0K Sep 26  2014 styles/
drwxr-xr-x 10 www-data root     4.0K Mar 12 19:24 tiles/
drwxr-xr-x  4 root     root     4.0K Feb  3  2015 warper/
drwxr-xr-x  9     1115 www-data 4.0K Dec 12  2017 wma/

Got clarification from @TheDJ via irc chat that the styles and tiles directories are the most concerning for him at the moment. I will focus on them and exclude chippy, warper, and wma.

Mentioned in SAL (#wikimedia-cloud) [2019-03-12T20:33:24Z] <bd808> Changing files owned by root to www-data:www-data in /data/project/{styles,tiles} from NFS server (T218145)

bd808 added a subscriber: Bstorm.Mar 12 2019, 8:37 PM

@Bstorm did some archeology for me and discovered that the root_squash was placed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/316713/ about 2.5 years ago. Unfortunately no associated task or comments to help understand why the change was made. It is however a good general practice to enforce root_squash on NFS exports which can be accessed by multiple user groups to help prevent various forms of attack (WP:BEANS).

bd808 added a comment.Mar 13 2019, 6:02 PM

This took about 20 hours to complete:

$ sudo ionice -c 3 nice -19 find /srv/maps/project/tiles -user root -exec chown www-data:www-data {} +
chown: cannot dereference ‘/srv/maps/project/tiles/mapnik-osm-carto-data/mapnik-osm-carto-data’: No such file or directory
chown: cannot dereference ‘/srv/maps/project/tiles/mapnik-osm-carto-data/data’:
Too many levels of symbolic links

@TheDJ I fear that there are some files left in need to ownership changes, but I think at this point you will need to find them one by one and ask for a Cloud root user to fix them for you.

bd808 closed this task as Resolved.Mar 14 2019, 5:50 AM

Resolving, but do reopen and add more files if you find them.