Page MenuHomePhabricator

Make scap and opcache work consistently together
Closed, ResolvedPublic

Description

PHP 7 opcache is how PHP stashes the bytecode cache of the files on disk. Given opcache works based on file paths and not inodes as APC did, it needs to constantly revalidate that its bytecode cache is valid as to avoid serving stale objects. This requires at least one stat() syscall, which is both quite expensive (in our sad post-SPECTRE world) alone, and probably a bottleneck for the future - we had a similar issue with HHVM, which has an admittedly smarter invalidation mechanism once you turn on stat_cache, that uses inotify amongst other things.

There are two approaches we can take to alleviate the number of stat syscalls from php-fpm:

  1. We decide to use a relatively short revalidation time (10 seconds) for the opcached files; this way we reduce the amount of time in which the code is in an inconsistent state. This approach might be dangerous during large releases (like the train releases).
  2. We disable opcache revalidation completely, and we wipe it at every release on all servers. I didn't check, but the performance impact of emptying the cache can be severe for a transient time (the actual impact still needs to be evaluated). For single-file releases, we could just invalidate in parallel all caches

The advantage of approach 1 is just... we don't need to modify scap for that to work. But I think the best course of action is the following:

  1. Extend the php admin interface we created to be able to invalidate single files using opcache_invalidate. This might be quite tricky.
  2. Have scap sync-file and sync-dir call /opcache-free?file=$arg on every appserver on the php7 admin port
  3. Have scap pull call php7adm /opcache-free locally
  4. Have scap sync run the above command in a rolling fashion after the code has been released everywhere

Of course canaries will need to be invalidated globally as soon as the code is released to them.

Event Timeline

Joe created this task.Dec 14 2018, 9:51 AM
Joe triaged this task as Normal priority.
jijiki added a subscriber: jijiki.Dec 14 2018, 10:01 AM
Joe updated the task description. (Show Details)Dec 19 2018, 8:40 AM

Change 480714 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php::monitoring: fine-grained opcache invalidation

https://gerrit.wikimedia.org/r/480714

Anomie removed a subscriber: Anomie.Dec 19 2018, 3:28 PM

Change 480714 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::php::monitoring: fine-grained opcache invalidation

https://gerrit.wikimedia.org/r/480714

Joe claimed this task.Jan 2 2019, 11:25 AM

(we'll watch via the Scap project)

Change 490888 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/tools/scap@master] Rewrite the concurrency logic of OpcacheManager

https://gerrit.wikimedia.org/r/490888

Change 490888 merged by jenkins-bot:
[mediawiki/tools/scap@master] Rewrite the concurrency logic of OpcacheManager

https://gerrit.wikimedia.org/r/490888

Change 493395 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] php::monitoring: allow deployment hosts to access the admin port

https://gerrit.wikimedia.org/r/493395

Change 493395 merged by Giuseppe Lavagetto:
[operations/puppet@production] php::monitoring: allow deployment hosts to access the admin port

https://gerrit.wikimedia.org/r/493395

GTirloni removed a subscriber: GTirloni.Feb 28 2019, 2:35 PM

Change 493418 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] php::monitoring: allow deployment hosts to reach the opcode endpoint

https://gerrit.wikimedia.org/r/493418

Change 493418 merged by Giuseppe Lavagetto:
[operations/puppet@production] php::monitoring: allow deployment hosts to reach the opcode endpoint

https://gerrit.wikimedia.org/r/493418

Change 493485 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] scap: fix php version, add php7 admin port

https://gerrit.wikimedia.org/r/493485

Change 493486 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php: stop revalidating opcache

https://gerrit.wikimedia.org/r/493486

Change 493485 merged by Giuseppe Lavagetto:
[operations/puppet@production] scap: fix php version, add php7 admin port

https://gerrit.wikimedia.org/r/493485

Change 493486 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::php: stop revalidating opcache

https://gerrit.wikimedia.org/r/493486

Joe closed this task as Resolved.Mar 4 2019, 9:44 AM