Parsoid-php is in a peculiar situation, where code deployed via scap3 is included into MediaWiki.
We have noticed that after a parsoid code deploy, the application servers were still serving an old version of the code.
This happens not because of an opcache malfunction (although the cure, probably, is the same in this case), but an underlying problem with how we include files and how opcache stores them.
When we store files in opcache, symbolic links are resolved. So when we include /srv/deployment/parsoid/src/$path, that becomes a cache key like /srv/deployment/parsoid/deploy-cache/revs/451db1e60149b05781ec1e63c949c932c08d4a8b/.
Sadly, php-fpm has a realpath cache that interferes with opcache invalidation, creating a race condition. See for instance https://engineering.facile.it/blog/eng/realpath-cache-is-it-all-php-opcache-s-fault/
I could prove it by simply watching the contents of opcache during a deployment:
while true; do sleep 1; date; TO_FIND=$(readlink /srv/deployment/parsoid/deploy); echo "Parsoid code at $TO_FIND"; php7adm opcache-meta; jq . /tmp/opcache_dump_meta | fgrep $TO_FIND - | wc -l; done
As I sadly expected, the realpath cache issue made the revalidation of opcache visit the old file that was never refreshed, with the exception of newly spawned children of php-fpm (as the realpath cache is process-local and not shared like opcache).
All concluded, I think we have the following options to solve the issue:
- Make scap3 restart php-fpm
- Disable the realpath cache on php-fpm
- Include parsoid-php in the mediawiki repository
I would opt for solution 2 temporarily, and move to 3 on the long term.