Page MenuHomePhabricator
Create Task
Maniphest T126658

/usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so no such file or directory
Closed, ResolvedPublic
Actions

Description

I have pooled six new Trusty instances a few hours ago and the HHVM jobs fail with:

 [Thu Feb 11 20:24:49 2016] [hphp] [14131:7f7c01f42d00:0:000001] []
Uncaught exception:
Could not open extension /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so:
  cannot open shared object file: No such file or directory

Slaves created previously got it though...

root@integration-saltmaster:~# salt -v '*slave-trusty*' cmd.run 'ls -l /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so'
Executing job with jid 20160211204006496675
-------------------------------------------

integration-slave-trusty-1006.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1001.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1002.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1004.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1003.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1016.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1018.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1012.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1017.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1005.integration.eqiad.wmflabs:
    ls: cannot access /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so: No such file or directory
integration-slave-trusty-1023.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1011.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1015.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1014.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
integration-slave-trusty-1013.integration.eqiad.wmflabs:
    -rw-r--r-- 1 root root 129408 Jun 15  2015 /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
root@integration-saltmaster:~#

The link does not belong to any package anymore:

integration-slave-trusty-1013:~$ dpkg -S /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so
dpkg-query: no path found matching pattern /usr/lib/x86_64-linux-gnu/hhvm/extensions/current/luasandbox.so

And we have:

$ apt-cache policy hhvm-luasandbox
hhvm-luasandbox:
  Installed: 2.0.11
  Candidate: 2.0.11
  Version table:
 *** 2.0.11 0
       1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy hhvm
hhvm:
  Installed: 3.6.5+dfsg1-1+wm8
  Candidate: 3.6.5+dfsg1-1+wm8
  Version table:
 *** 3.6.5+dfsg1-1+wm8 0
       1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

Related Objects
Search...

Event Timeline

hashar created this task.Feb 11 2016, 8:43 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added projects: Continuous-Integration-Infrastructure, SRE, HHVM.
hashar added subscribers: hashar, Catrope.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 11 2016, 8:43 PM
Paladox set Security to None.Feb 11 2016, 8:45 PM
Paladox subscribed.
hashar added a comment.Feb 11 2016, 8:52 PM

The link is provided by puppet. modules/hhvm/manifests/init.pp has:

hhvm         => {
    dynamic_extension_path   => '/usr/lib/x86_64-linux-gnu/hhvm/extensions/current',
    dynamic_extensions       => [ 'luasandbox.so', 'tidy.so', 'wikidiff2.so' ],

And the link is maintained by the hhvm upstart service directly:

pre-start script
  ...
  # Update the symlink `/usr/lib/x86_64-linux-gnu/hhvm/extensions/current`
  # so it points to the directory whose name matches the extension API
  # version (e.g. `/usr/lib/x86_64-linux-gnu/hhvm/extensions/20140829`).
  API_VERSION="$(/usr/bin/hhvm --version | grep -Po '(?<=API: ).+')"
  ( cd /usr/lib/x86_64-linux-gnu/hhvm/extensions; ln -Trsf "$API_VERSION" current; )
end script

On CI, I have flagged the hhvm service to not run with https://gerrit.wikimedia.org/r/#/c/269947/ ( T126594 ) because we have no use for it.

Shouldn't that be handled by the .deb package directly?

hashar added a parent task: T126594: Disable HHVM fcgi server on CI slaves.Feb 11 2016, 9:02 PM
hashar moved this task from Untriaged to Externally Blocked on the Continuous-Integration-Infrastructure board.Feb 18 2016, 11:05 PM
greg subscribed.Feb 19 2016, 12:42 AM

@hashar since this is externally blocked for us, would that mean it's blocked on ops? That my best guess here. (Confirming before I unilaterally add that project/tag.)

hashar added a project: Blocked-on-Operations.Feb 19 2016, 3:15 PM

I am always assuming a task having SRE implies it is Blocked-on-Operations . Made that explicit.

For ops / HHVM folks the summary is:

  • hhvm extensions are build to install under /usr/lib/x86_64-linux-gnu/hhvm/extensions/$HHVM_API_VERSION
  • config files use /current
  • hhvm upstart service on startup create the /current --> $HHVM_API_VERSION symlink

When the HHVM service is not enabled, the symlink current is not created and the HHVM CLI fail loading extensions because it can't find them under /current.

We would need the HHVM package to handle the symlink creation/renaming instead of upstart dealing with it.

• ema triaged this task as Medium priority.Feb 23 2016, 9:20 AM
Joe subscribed.Feb 23 2016, 9:23 AM

@hashar no we definitely don't want the debian package to deal with this hack we've come up with. We should anyways create "current" via puppet directly.

akosiaris subscribed.Apr 20 2016, 9:04 AM

I see rOPUPf6c222149 has been merged since which might have fixed this. @hashar does the problem still exist?

hashar closed this task as Resolved.Apr 20 2016, 1:30 PM
hashar assigned this task to ori.
hashar added a subscriber: ori.

Indeed @ori fixed it. On a CI slave I can confirm that /usr/lib/x86_64-linux-gnu/hhvm/extensions/current is no more created. The config files having:

# grep linux-g /etc/hhvm/*.ini
/etc/hhvm/fcgi.ini:hhvm.dynamic_extension_path = /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212
/etc/hhvm/php.ini:hhvm.dynamic_extension_path = /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212
/etc/hhvm/server.ini:hhvm.dynamic_extension_path = /usr/lib/x86_64-linux-gnu/hhvm/extensions/20150212

So that got solved by having puppet to hardcode the HHVM API version (20150212) in the various .ini files. Thanks!

Stashbot subscribed.Apr 20 2016, 1:33 PM

Mentioned in SAL [2016-04-20T13:33:02Z] <hashar> salt -v '*trusty*' cmd.run 'rm /usr/lib/x86_64-linux-gnu/hhvm/extensions/current' # Cleanup on CI slaves for T126658

hashar mentioned this in T126594: Disable HHVM fcgi server on CI slaves.Apr 20 2016, 1:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL