The admin module of HHVM has quite a few diagnostics on the status of the memory usage in hhvm; So what I found on one of the affected machines, where hhvm were occupying 10 gigabytes of memory, while allocating 27 gb of virtual memory:

• APC is using a significant yet small amount of available memory
• the PCRE cache is less than 100 Mb
• the TC cache has 870 Mb dedicated to it and it's heavilly underused
• Static Strings take up around 100 Mb

and there is not really any common outlier to justify this memory consumption.

I can confirm that with heap profiling activated HHVM crashes too often to make collecting heap diffs significant.

• APC is using a significant yet small amount of available memory

Do we need to reduce the amount of things we're putting into APC? Or increase how much APC space there is..?

Is this the same as the recent behavior seen on various API appservers? https://ganglia.wikimedia.org/latest/?c=API%20application%20servers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2

Some of the older appservers (with 32 threads) have an average CPU usage of >80%, where the high increase started around September 7.. Memory leaks are also present here.

Mentioned in SAL (#wikimedia-operations) [2016-09-17T16:40:48Z] <_joe_> rolling restart of HHVM on part fo the API cluster in eqiad, T133674