/usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so no such file or directory
Closed, ResolvedPublic

Description

I have pooled six new Trusty instances a few hours ago and the HHVM jobs fail with:

 [Thu Feb 11 20:24:49 2016] [hphp] [14131:7f7c01f42d00:0:000001] []
Uncaught exception:
Could not open extension /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so:
  cannot open shared object file: No such file or directory

Slaves created previously got it though...

root@integration-saltmaster:~# salt -v '*slave-trusty*' cmd.run 'ls -l /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so'
Executing job with jid 20160211204006496675
-------------------------------------------

integration-slave-trusty-1006.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1001.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1002.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1004.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1003.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1016.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1018.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1012.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1017.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1005.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1023.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1011.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1015.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1014.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1013.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
root@integration-saltmaster:~#

The link does not belong to any package anymore:

integration-slave-trusty-1013:~$ dpkg -S /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
dpkg-query: no path found matching pattern /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so

And we have:

$ apt-cache policy hhvm-luasandbox
hhvm-luasandbox:
  Installed: 2.0.11
  Candidate: 2.0.11
  Version table:
 *** 2.0.11 0
       1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy hhvm
hhvm:
  Installed: 3.6.5+dfsg1-1+wm8
  Candidate: 3.6.5+dfsg1-1+wm8
  Version table:
 *** 3.6.5+dfsg1-1+wm8 0
       1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 11 2016, 8:43 PM
Paladox set Security to None.Feb 11 2016, 8:45 PM
Paladox added a subscriber: Paladox.

The link is provided by puppet. modules/hhvm/manifests/init.pp has:

hhvm         => {
    dynamic_extension_path   => '/usr/lib/x86_64-linux-gnu/hhvm/extensions/current',
    dynamic_extensions       => [ 'luasandbox.so', 'tidy.so', 'wikidiff2.so' ],

And the link is maintained by the hhvm upstart service directly:

pre-start script
  ...
  # Update the symlink `/usr/lib/x86_64-linux-gnu/hhvm/extensions/current`
  # so it points to the directory whose name matches the extension API
  # version (e.g. `/usr/lib/x86_64-linux-gnu/hhvm/extensions/20140829`).
  API_VERSION="$(/usr/bin/hhvm --version | grep -Po '(?<=API: ).+')"
  ( cd /usr/lib/x86_64-linux-gnu/hhvm/extensions; ln -Trsf "$API_VERSION" current; )
end script

On CI, I have flagged the hhvm service to not run with https://gerrit.wikimedia.org/r/#/c/269947/ ( T126594 ) because we have no use for it.

Shouldn't that be handled by the .deb package directly?

greg added a subscriber: greg.Feb 19 2016, 12:42 AM

@hashar since this is externally blocked for us, would that mean it's blocked on ops? That my best guess here. (Confirming before I unilaterally add that project/tag.)

I am always assuming a task having Operations implies it is Blocked-on-Operations . Made that explicit.

For ops / HHVM folks the summary is:

  • hhvm extensions are build to install under /usr/lib/x86_64-linux-gnu/hhvm/extensions/$HHVM_API_VERSION
  • config files use /current
  • hhvm upstart service on startup create the /current --> $HHVM_API_VERSION symlink

When the HHVM service is not enabled, the symlink current is not created and the HHVM CLI fail loading extensions because it can't find them under /current.

We would need the HHVM package to handle the symlink creation/renaming instead of upstart dealing with it.

ema triaged this task as "Normal" priority.Feb 23 2016, 9:20 AM
Joe added a subscriber: Joe.Feb 23 2016, 9:23 AM

@hashar no we definitely don't want the debian package to deal with this hack we've come up with. We should anyways create "current" via puppet directly.

I see rOPUPf6c222149 has been merged since which might have fixed this. @hashar does the problem still exist?

hashar assigned this task to ori.Apr 20 2016, 1:30 PM
hashar closed this task as "Resolved".
hashar added a subscriber: ori.

Indeed @ori fixed it. On a CI slave I can confirm that /usr/lib/x86_64-linux-gnu/hhvm/extensions/current is no more created. The config files having:

# grep linux-g /etc/hhvm/*.ini
/etc/hhvm/fcgi.ini:hhvm.dynamic_extension_path = /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212
/etc/hhvm/php.ini:hhvm.dynamic_extension_path = /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212
/etc/hhvm/server.ini:hhvm.dynamic_extension_path = /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212

So that got solved by having puppet to hardcode the HHVM API version (20150212) in the various .ini files. Thanks!

Mentioned in SAL [2016-04-20T13:33:02Z] <hashar> salt -v '*trusty*' cmd.run 'rm /usr/lib/x86_64-linux-gnu/hhvm/extensions/current' # Cleanup on CI slaves for T126658