PHP 7 opcache is how PHP stashes the bytecode cache of the files on disk. Given opcache works based on file paths and not inodes as APC did, it needs to constantly revalidate that its bytecode cache is valid as to avoid serving stale objects. This requires at least one `stat()` syscall, which is both quite expensive (in our sad post-SPECTRE world) alone, and probably a bottleneck for the future - we had a similar issue with HHVM, which has an admittedly smarter invalidation mechanism once you turn on `stat_cache`, that uses inotify amongst other things.
There are two approaches we can take to alleviate the number of stat syscalls from php-fpm:
# We decide to use a relatively short revalidation time (10 seconds) for the opcached files; this way we reduce the amount of time in which the code is in an inconsistent state. This approach might be dangerous during large releases (like the train releases).
# We disable opcache revalidation completely, and we wipe it at every release on all servers. I didn't check, but the performance impact of emptying the cache can be severe for a transient time (the actual impact still needs to be evaluated). For single-file releases, we could just invalidate in parallel all caches
The advantage of approach 1 is just... we don't need to modify scap for that to work. But I think the best course of action is the following:
# Extend the php admin interface we created to be able to invalidate single files using `opcache_invalidate`. This might be quite tricky.
# Have scap sync-file and sync-dir call `php7adm /opcache-invalidate-last` on every appserver (or call the corresponding url remotely)
# Have scap pull call `php7adm /opcache-free`
# Have scap sync run the above command in a rolling fashion after the code has been released everywhere
Of course canaries will need to be invalidated globally as soon as the code is released to them.