The static server often returns HTTP 403 Forbidden for files recently updated from Git. This seems to be an NFS issue based on discussion with valhallasw in Cloud-Services.

Known facts:

• This happens even if I break the symlink from ~/www/static to the git repository before updating.
• This happens very reliably, but only for modified files. (Updating the repository doesn't break files that didn't change.)
• The 403 seems to be cached on the server; after renaming the file, the new name works fine but the old name keeps returning 403 (even though it's not there anymore).
• This seems to last for quite a while (5+ hours) until it starts the serving the file.
• Remounting the NFS share on the static server solves the problem temporarily (including preventing further occurrences), but it eventually started happening again.

When I renamed an affected file (so both names below should return 'no such file'), valhallasw noticed this behaviour from the static server accessing the NFS share:

valhallasw@tools-web-static-01:/data/project/meta/www/static/scripts/edge$ cat pathoschild.regexeditor.css
cat: pathoschild.regexeditor.css: Permission denied
valhallasw@tools-web-static-01:/data/project/meta/www/static/scripts/edge$ cat pathoschild.regexeditor.csss
cat: pathoschild.regexeditor.csss: No such file or directory

Repro steps:

1. Become a Tool Labs project.
2. Clone a git repository containing CSS/JS files to ~/git/<repo>.
3. Symlink ~/www/static/<repo> to ~/git/<repo>.
4. View one of the files via tools-static.wmflabs.org/<project name>/<repo>/<filename> (you should get HTTP 200 OK).
5. Perform an upstream change to one of the files.
6. Update the git repo (git pull).
7. View the same file via tools-static.wmflabs.org (you should get HTTP 403 Forbidden).