@Joe @ArielGlenn I'm depending on SRE for these metrics to exist and be graphed. The second step of evaluating them and looking for any low-hanging fruit (in MediaWiki) is something you can move the task back to our Inbox for.

Hi @Krinkle the metrics for php7 exist already, they're exported to prometheus as follows:

# HELP php_apcu_num_slots Number of distinct APCu slots available
# TYPE php_apcu_num_slots counter
php_apcu_num_slots 4099
# HELP php_apcu_cache_ops Stats about APCu operations
# TYPE php_apcu_cache_ops counter
php_apcu_cache_ops{type="hits"} 6148
php_apcu_cache_ops{type="misses"} 6148
php_apcu_cache_ops{type="inserts"} 2899
php_apcu_cache_ops{type="entries"} 2584
php_apcu_cache_ops{type="expunges"} 17974
# HELP php_apcu_memory APCu memory status
# TYPE php_apcu_memory gauge
php_apcu_memory{type="free"} 1258712
php_apcu_memory{type="total"} 33554312

the operations rate per time unit is already graphed there. There is more information we might be interested in, but to extract that we just need to modify the php code that offers the prometheus metrics under modules/profile/files/mediawiki/php/admin in operations/puppet. Looking at graphs for apcu free memory now, it looks like we need a much larger apc cache now that we're actually sending traffic to php7, by the way.

A couple general graphs were added to

https://grafana.wikimedia.org/d/GuHySj3mz/php7-transition?refresh=30s&orgId=1&from=now-12h&to=now

to monitor APC status.

I would argue we might want to raise the memory dedicated to APCu a bit more and see if that improves the cache-hit ratio, which is around 85% on average right now.

@Joe The cache hit ratio and rate per second aren't metrics we previously monitored for HHVM. In retrospect, I suppose that would've been useful. I was hoping we'd have something comparable to HHVM APC Usage where we plot the total value size.

On the other hand, for HHVM the size was effectively unbounded (which made size a useful thing to check, e.g. to detect if MW forgets to set a TTL and builds up stale keys). For PHP this may be less useful, assuming it simply uses up all available space.

Anyway, I've added a panel to PHP7 transition that plots total - free, just so that we have it for now. It shows most servers using ~ 500 MB which makes sense given the limit of 512 MB we just set.

By comparison, the HHVM dash shows servers generally use a stable ~ 4 GB space. That's quite a big difference. It's possible most of that is keys that will never be used. On the other hand, I'm not aware of anything using indeterministic keys for APCu (which would be a bug). So, presumably reducing it to anything lower than 4 GB would increase cache misses and increase computations needed. That might be acceptable, but might also be something we could tune down later if memory is an issue. Can we afford another 4 GB on app servers during the transition?

Looking at ResourceLoader for potential impact during the current roll out. The main use of APCu there is for the minification cache. No definitive answers there, but a quick scan does show a potential correlation with PHP 7 deployments.

ResourceLoader / minify cache hit ratio:

• minify-css shows a drop from a long-term 87-90% down to 92% on 2019-05-01. It climbed back on 2019-05-21.
• minify-js shows a drop from a long-term 95-97% down to 92% on 2019-05-01. It climbed back on 2019-05-21.
• p95 build time for 1 chunk increased from well under 50ms (usually ~3ms) to consistently between 70ms and 115 ms. The ~3ms spots started coming back at 2019-05-21 06:59, with the last 100ms bump going away at 2019-05-21 11:08.

Server Admin Log:

2019-03-25 12:08 Move 0.1% of anonymous users to php7 T212828
2019-04-30 15:09: Send 1% of anonymous users to PHP7.2 - T219150
2019-05-02 08:55: Send 5% of anonymous users to PHP7.2 - T219150
2019-05-21 06:59: Turning off php7 sampling for investigation in T223952
2019-05-21 10:59 _joe_: rolling restart of php7.2-fpm across the fleet to pick up a config change
2019-05-21 11:27: Revert "Switch off php7 for investigation of production instabilities" (enable for 5%)
2019-05-21 15:37: Moving to 10% of users on php7 T219150

This could be a coincidence of course, but worth looking into.

@Krinkle very interesting data about the resourceloader performance, I think it's no coincidence. More on that below.

Coming to APCu: it's a very different beast than how APC works in HHVM. It is strongly bound to the top resource usage we set in apc.shm_size, but it also runs a GC thread every hour that removes all expired keys (this is also active in HHVM), then there is a special LRU-based GC that happens when the memory gets full.

I wanted intentionally to grow progressively across a week or so the size of the shm size, now that we have some meaningful amount of traffic going to php-fpm. My intention is to raise it in steps and see how that affects the cache hit ratio.

Coming to the graph with the total number of items in apc - we don't export that metric currently but we can work on the php code that exports the prometheus metrics, which is under operations/puppet:module/profile/files/mediawiki/php. I'll try to get to it but contributions welcome (and also, suggestions on what else to monitor).

Looking better at the last graph you pasted, I don't really explain myself why the p95 went down so much when we reintroduced php7 at 10% with respect to before we reenabled it. In theory HHVM should drive those metrics in that situation?

Change 512118 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::mediawiki::php: make apc size configurable, bump for appservers

https://gerrit.wikimedia.org/r/512118

Change 512118 merged by Giuseppe Lavagetto:
[operations/puppet@production] profile::mediawiki::php: make apc size configurable, bump for appservers

https://gerrit.wikimedia.org/r/512118

Mentioned in SAL (#wikimedia-operations) [2019-05-23T10:15:32Z] <_joe_> restarted php7.2-fpm on mw1261 to assess the effect of a larger APCu shm size T223180

After doubling the cache size on one server, I noticed the cache-hit ratio plateaued between 80% and 90% after ~ 150 MB of space were occupied. I'll let it grow more, but if this is the case, I think we should aim at keeping the APC memory smaller and thus having smaller LRU flushes more frequently.

I'm rolling back the size of APCu to 512M after seeing how, on mw1261, the occupied memory grew significantly but the cache hit ratio didn't. IMHO it only risks to create a larger flush of LRU events and thus more disruption.

Mentioned in SAL (#wikimedia-operations) [2019-05-27T09:52:20Z] <_joe_> disabling puppet on mw1261, running some tests for T223180