The HHVM servers in the API pool show high system CPU% usage, with peak values that are consistent with one or more cores being maxed (12.5% and 25% are common). Zend app servers handling the same load on similar hardware specs rarely go above 5%.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
I collected system call counts with perf on Zend and HHVM servers and diffed them. I was not sure how to get time-per-call stats from perf, but the counts are still interesting: readlink, lseek and newlstat are the standouts.
While poking around the HHVM source tree, I came across the hhvm.server.stat_cache runtime option, which enables StatCache, which caches stat calls, using inotify to invalidate. I enabled it on mw1081 and system CPU% was cut nearly in half.
The next step is to test it more thoroughly to verify that cache invalidation happens promptly and reliably when the files are touched.