The static server often returns HTTP 403 Forbidden for files recently updated from Git. This seems to be an NFS issue based on discussion with valhallasw in #wikimedia-labs.
**Known facts:**
* This happens even if I break the symlink from `~/www/static` to the git repository before updating.
* This happens very reliably, but only for modified files. (Updating the repository doesn't break files that didn't change.)
* The 403 seems to be cached on the server; after renaming the file, the new name works fine but the old name keeps returning 403 (even though it's not there anymore).
* This seems to last for quite a while (5+ hours) until it starts the serving the file.
* Remounting the NFS share on the static server solves the problem temporarily (including preventing further occurrences), but it eventually started happening again.
When I renamed an affected file (so both names below should return 'no such file'), valhallasw noticed this behaviour from the static server accessing the NFS share:
valhallasw@tools-web-static-01:/data/project/meta/www/static/scripts/edge$ cat pathoschild.regexeditor.css
cat: pathoschild.regexeditor.css: Permission denied
valhallasw@tools-web-static-01:/data/project/meta/www/static/scripts/edge$ cat pathoschild.regexeditor.csss
cat: pathoschild.regexeditor.csss: No such file or directory
**Repro steps:**
# Become a Tool Labs project.
# Clone [a git repository containing CSS/JS files](https://github.com/Pathoschild/Wikimedia-contrib) to `~/git/<repo>`.
# Symlink `~/www/static/<repo>` to `~/git/<repo>`.
# View one of the files via `tools-static.wmflabs.org/<project name>/<repo>/<filename>` (you should get HTTP 200 OK).
# Perform an upstream change to one of the files.
# Update the git repo (`git pull`).
# View the same file via tools-static.wmflabs.org (you should get HTTP 403 Forbidden).