Page MenuHomePhabricator

Class 'Memcached' not found for php7 in beta
Closed, ResolvedPublic

Description

Call to mwscript eval.php returned: None (duration: 00m 00s)

13:25:42 Checking for new runtime errors locally
13:25:42 Scap failed!: Call to mwscript eval.php returned: None
13:25:42 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 342, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/main.py", line 603, in main
    return super(Scap, self).main(*extra_args)
  File "/usr/lib/python2.7/dist-packages/scap/main.py", line 80, in main
    self._check_fatals()
  File "/usr/lib/python2.7/dist-packages/scap/main.py", line 217, in _check_fatals
    raise RuntimeError(errmsg.format('returned', exit_code))
RuntimeError: Scap failed!: Call to mwscript eval.php returned: None
13:25:42 scap failed: RuntimeError Scap failed!: Call to mwscript eval.php returned: None (duration: 00m 00s)

See also: T217539: scap in job beta-scap-eqiad has random failures: eval.php cannot find group 50062

Event Timeline

hashar created this task.Mar 4 2019, 6:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2019, 6:09 PM
hashar added a comment.Mar 4 2019, 6:12 PM

I am not quite sure what is happening here. It comes from AbstractSync._check_fatals which unfortunately shallows the mwscript exception and return its own exception. Maybe giving more details as to why it fails would be informative.

I have no idea why the exit code would be None. I looked a bit at scap/sh.py but it is rather complicated :-\

mmodell added a subscriber: mmodell.Mar 4 2019, 6:18 PM

I am not quite sure what is happening here. It comes from AbstractSync._check_fatals which unfortunately shallows the mwscript exception and return its own exception. Maybe giving more details as to why it fails would be informative.
I have no idea why the exit code would be None. I looked a bit at scap/sh.py but it is rather complicated :-\

The problem is actually in mwscript, not in scap ( T121597#4681823 explains some background )

I'm not sure why it's showing None but the actual problem is:

[thcipriani@deployment-deploy01 ~]$ echo '1' | PHP=php7.2 mwscript eval.php --wiki=enwiki                                                             
Fatal error: Uncaught Error: Class 'Memcached' not found in /srv/mediawiki-staging/php-master/includes/libs/objectcache/MemcachedPeclBagOStuff.php:61 
Stack trace:
#0 /srv/mediawiki-staging/php-master/includes/objectcache/ObjectCache.php(216): MemcachedPeclBagOStuff->__construct(Array)                            
#1 /srv/mediawiki-staging/php-master/includes/objectcache/ObjectCache.php(338): ObjectCache::newFromParams(Array)                                     
#2 /srv/mediawiki-staging/php-master/includes/ServiceWiring.php(262): ObjectCache::newWANCacheFromParams(Array)                                       
#3 /srv/mediawiki-staging/php-master/includes/libs/services/ServiceContainer.php(430): Wikimedia\Services\ServiceContainer->{closure}(Object(MediaWiki\MediaWikiServices))
#4 /srv/mediawiki-staging/php-master/includes/libs/services/ServiceContainer.php(414): Wikimedia\Services\ServiceContainer->createService('MainWANObjectCa...')
#5 /srv/mediawiki-staging/php-master/includes/MediaWikiServices.php(650): Wikimedia\Services\ServiceContainer->getService('MainWANObjectCa...')       
#6 /srv/mediawiki-staging/php-master/includes/db/M in /srv/mediawiki-staging/php-master/includes/libs/objectcache/MemcachedPeclBagOStuff.php on line 61

Although php7.2-memcached seems installed:

[thcipriani@deployment-deploy01 ~]$ apt-cache policy php-memcached
php-memcached:
  Installed: 3.0.1+2.2.0-1~wmf1
  Candidate: 3.0.1+2.2.0-1~wmf1
  Version table:
     3.0.1+2.2.0-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
 *** 3.0.1+2.2.0-1~wmf1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/thirdparty/php72 amd64 Packages
        100 /var/lib/dpkg/status
hashar added a subscriber: Joe.Mar 5 2019, 4:02 PM

beta-scap-eqiad constantly fails now. The last good build was today March 5th at 11:43:02

On deployment-deploy01 scap got upgraded but that does not seem related

Start-Date: 2019-03-05  11:42:54
Commandline: apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install scap=3.9.2-1
Upgrade: scap:amd64 (3.9.1-1, 3.9.2-1)
End-Date: 2019-03-05  11:42:56

Still from /var/log/apt/history.log, some php packages have been removed/installed by @Joe :

Start-Date: 2019-03-01  06:58:43
Commandline: apt-get remove --purge php-memcached
Requested-By: oblivian (4816)
Purge: php-memcached:amd64 (3.0.1+2.2.0-1)
End-Date: 2019-03-01  06:58:45
Start-Date: 2019-03-01  06:59:32
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php-memcached
Requested-By: oblivian (4816)
Install: php-memcached:amd64 (3.0.1+2.2.0-1~wmf1)
End-Date: 2019-03-01  06:59:33
thcipriani triaged this task as High priority.Mar 5 2019, 7:16 PM
thcipriani added a subscriber: MoritzMuehlenhoff.

I have disabled beta-scap-eqiad since it's now failing constantly.

Other clues:

Noticed this this morning, which is strange:

$ strace -ofile -f php7.2 -i
open("/usr/lib/php/20170718/memcached.so", O_RDONLY|O_CLOEXEC) = 4                                                                                    
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)                                                                       
open("/usr/lib/x86_64-linux-gnu/libmemcached.so.11", O_RDONLY|O_CLOEXEC) = 4                                                                          
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)                                                                       
open("/usr/lib/x86_64-linux-gnu/libmemcachedutil.so.2", O_RDONLY|O_CLOEXEC) = 4                                                                       
open("/usr/lib/php/20170718/memcached.so.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 

Still from /var/log/apt/history.log, some php packages have been removed/installed by @Joe :

Start-Date: 2019-03-01  06:58:43
Commandline: apt-get remove --purge php-memcached
Requested-By: oblivian (4816)
Purge: php-memcached:amd64 (3.0.1+2.2.0-1)
End-Date: 2019-03-01  06:58:45
Start-Date: 2019-03-01  06:59:32
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php-memcached
Requested-By: oblivian (4816)
Install: php-memcached:amd64 (3.0.1+2.2.0-1~wmf1)
End-Date: 2019-03-01  06:59:33

This may be related to a change by @MoritzMuehlenhoff from a few hours ago: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494212/

Mentioned in SAL (#wikimedia-releng) [2019-03-05T19:17:26Z] <thcipriani> disable beta-scap-eqiad due to T217587

thcipriani renamed this task from scap in job beta-scap-eqiad has random failures: Call to mwscript eval.php returned: None to Class 'Memcached' not found for php7 in beta.Mar 5 2019, 7:19 PM
Joe added a comment.Mar 5 2019, 8:06 PM

apt-get install php7.2-msgpack solved the problem FWIW.

Joe closed this task as Resolved.Mar 5 2019, 8:06 PM
Joe claimed this task.
hashar added a comment.Mar 6 2019, 8:23 AM

And I think the php package fixed eventually caused another issue with curl when running update.php T217733