Change 190771 had a related patch set uploaded (by BryanDavis):
Set file permissions for hhbc cache files

https://gerrit.wikimedia.org/r/190771

Patch-For-Review

Change 190771 merged by jenkins-bot:
Set file permissions for hhbc cache files

https://gerrit.wikimedia.org/r/190771

In T89390#1081029, @bd808 wrote:

I think the "fix" I made only takes effect when vagrant provision is run after the hhbc files have been created with improper permissions. This may be caused by HHVM deleting the empty touched files and replacing them with proper sqlite files.

Not necessarily. To clarify my comment in the patch you mention, no vagrant provision is ever used. With

VAGRANT_DEFAULT_PROVIDER=lxc vagrant up

hhvm files have the wrong owner/permissions combo (but not the dir). After that, doing:

vagrant destroy
vagrant up

creates the vbox version with the right owner/permissions from the start.

Ok, this is rather peculiar. After 3 or 4 iterations of up + destroy, the issue seems to have disappeared for LXC. I don't understand what's going on here.

Some possibly pertinent info: while the issue persisted, there were no differences in provisioning logs between LXC and VBox. Furthermore, there are no apparent version differences between the two boxes, apart from puppet, but the change is a minor one: 3.4.3-1 in VBox vs 3.4.3-1ubuntu1 in LXC.

In T89390#1081029, @bd808 wrote:

This may be caused by HHVM deleting the empty touched files and replacing them with proper sqlite files.

Maybe what's happening is that sometimes puppet manages to touch the files before hhvm creates them, while other times hhvm creates the files first, and then pupper checks for their owner/perms? This would explain the different outcomes we are witnessing.

I doubt this non-determinism is possible, but throwing the idea out there as food for thought.

In T89390#1081814, @mobrovac wrote:

In T89390#1081029, @bd808 wrote:

This may be caused by HHVM deleting the empty touched files and replacing them with proper sqlite files.

Maybe what's happening is that sometimes puppet manages to touch the files before hhvm creates them, while other times hhvm creates the files first, and then pupper checks for their owner/perms? This would explain the different outcomes we are witnessing.

I doubt this non-determinism is possible, but throwing the idea out there as food for thought.

Actually I think it is quite possible. Puppet operates by creating a directed acyclic graph of operations to perform on the system. In the absence of direct dependency information between nodes in that graph it uses random numbers to decide where to put things. In ::mediawiki we include ::hhvm which tells puppet to do all the things declared in ::hhvm and it's children before doing things that depend on ::mediawiki but the ordering is not any more specific than that. It would be quite possible for a given Puppet run to do (install hhvm -> run a script -> chmod files) in one run and (chmod files -> install hhvm -> run a script) in another.

Changing from include ::hhvm to require ::hhvm would make this a bit more deterministic by telling Puppet that it must do all the ::hhvm operations and their dependencies before doing anything in ::mediawiki. That alone may not fix the problem but it should make the result of the first Puppet run triggered by vagrant up more deterministic. I'll test with some local patches and see if I can narrow things down.

Change 194147 had a related patch set uploaded (by BryanDavis):
Ensure that hhvm is fully installed before mediawiki

https://gerrit.wikimedia.org/r/194147

Change 194147 merged by Mobrovac:
Ensure that hhvm is fully installed before mediawiki

https://gerrit.wikimedia.org/r/194147