Just to confirm: happens for me, too, I jsut wanted to test something (;)) and I was wondering why all actions (rendering of existing page and/or parsing of an edit, both on a mobile phone) takes so long :/
Page generation times seem to be getting better and better for testwiki. @ori had a reasonable explanation that this is due to the 11+ hits per server needed to fully warm up the HHVM JIT system. 1.27.0-wmf.12 is the first new branch to be deployed where testwiki is being served by the general MediaWiki web server pool rather than being exclusively handled by wm1017.eqiad.wmnet. With the small load going to testwiki being spread over a much larger pool of servers it will take longer for the full pool to have warm caches.
note we want to update our deployment train doc to have the whole group0 switched instead of just testwiki. https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys
That would warm up the cache faster still with a limited scope (only group0).
Changing priority, no more an issue. I basically freaked out :-}
This was from expect HHVM cache warming. Full IRC log below for those who weren't there:
23:18 < hashar> so hhvm warming up its cache ? 23:18 < ori> JIT threshold + few users + many app servers = high likehood of slow request 23:18 < ori> yes 23:19 < hashar> so that would only happen when we roll a new branch? 23:19 < ori> there's what, a half dozen of us making requests to test and test2, plus a handful of random users 23:19 < ori> the requests that we're making are distributed over 200+ app servers 23:19 < ori> so there's a high chance that your request is hitting an app server that is translating the code in the wmf12 branch for the first time 23:19 < bd808> it does seem to be getting better and better 23:20 < hashar> is a single hit enough to fully populate the jit / cache whatever? 23:20 < ori> no, it's something like 11 IIRC 23:20 < bd808> first hit would prime the apc cache equivalent. it takes 11+ to fully warm up the JIT 23:20 < hashar> so by only deploying to testwiki , that hasn't attracted enough traffic 23:20 < hashar> and I freaked out 23:21 < ori> Yes, I think so. I freaked out too, but I think this is what happened. 23:21 < hashar> whereas had I deployed to mediawikiwiki that would have populated much faster 23:21 < ori> yep. 23:21 < bd808> this is our first full scap wiht testwiki distributed to the cluster too correct? 23:21 < ori> first full branch maybe; I think there have been other scaps ... 23:23 < ori> at any rate, I think it is fine to roll out wmf12 to all group0 wikis