Page MenuHomePhabricator
Create Task
Maniphest T217587

Class 'Memcached' not found for php7 in beta
Closed, ResolvedPublic
Actions

Description

Call to mwscript eval.php returned: None (duration: 00m 00s)

13:25:42 Checking for new runtime errors locally
13:25:42 Scap failed!: Call to mwscript eval.php returned: None
13:25:42 Unhandled error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 342, in run
    exit_status = app.main(app.extra_arguments)
  File "/usr/lib/python2.7/dist-packages/scap/main.py", line 603, in main
    return super(Scap, self).main(*extra_args)
  File "/usr/lib/python2.7/dist-packages/scap/main.py", line 80, in main
    self._check_fatals()
  File "/usr/lib/python2.7/dist-packages/scap/main.py", line 217, in _check_fatals
    raise RuntimeError(errmsg.format('returned', exit_code))
RuntimeError: Scap failed!: Call to mwscript eval.php returned: None
13:25:42 scap failed: RuntimeError Scap failed!: Call to mwscript eval.php returned: None (duration: 00m 00s)

See also: T217539: scap in job beta-scap-eqiad has random failures: eval.php cannot find group 50062

Related Objects

Event Timeline

hashar created this task.Mar 4 2019, 6:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2019, 6:09 PM
hashar mentioned this in T217539: scap in job beta-scap-eqiad has random failures: eval.php cannot find group 50062.Mar 4 2019, 6:09 PM
hashar updated the task description. (Show Details)
hashar added a comment.Mar 4 2019, 6:12 PM

I am not quite sure what is happening here. It comes from AbstractSync._check_fatals which unfortunately shallows the mwscript exception and return its own exception. Maybe giving more details as to why it fails would be informative.

I have no idea why the exit code would be None. I looked a bit at scap/sh.py but it is rather complicated :-\

• mmodell subscribed.Mar 4 2019, 6:18 PM
In T217587#4999010, @hashar wrote:

I am not quite sure what is happening here. It comes from AbstractSync._check_fatals which unfortunately shallows the mwscript exception and return its own exception. Maybe giving more details as to why it fails would be informative.

I have no idea why the exit code would be None. I looked a bit at scap/sh.py but it is rather complicated :-\

The problem is actually in mwscript, not in scap ( T121597#4681823 explains some background )

thcipriani added a comment.Mar 5 2019, 1:02 PM

I'm not sure why it's showing None but the actual problem is:

[thcipriani@deployment-deploy01 ~]$ echo '1' | PHP=php7.2 mwscript eval.php --wiki=enwiki                                                             
Fatal error: Uncaught Error: Class 'Memcached' not found in /srv/mediawiki-staging/php-master/includes/libs/objectcache/MemcachedPeclBagOStuff.php:61 
Stack trace:
#0 /srv/mediawiki-staging/php-master/includes/objectcache/ObjectCache.php(216): MemcachedPeclBagOStuff->__construct(Array)                            
#1 /srv/mediawiki-staging/php-master/includes/objectcache/ObjectCache.php(338): ObjectCache::newFromParams(Array)                                     
#2 /srv/mediawiki-staging/php-master/includes/ServiceWiring.php(262): ObjectCache::newWANCacheFromParams(Array)                                       
#3 /srv/mediawiki-staging/php-master/includes/libs/services/ServiceContainer.php(430): Wikimedia\Services\ServiceContainer->{closure}(Object(MediaWiki\MediaWikiServices))
#4 /srv/mediawiki-staging/php-master/includes/libs/services/ServiceContainer.php(414): Wikimedia\Services\ServiceContainer->createService('MainWANObjectCa...')
#5 /srv/mediawiki-staging/php-master/includes/MediaWikiServices.php(650): Wikimedia\Services\ServiceContainer->getService('MainWANObjectCa...')       
#6 /srv/mediawiki-staging/php-master/includes/db/M in /srv/mediawiki-staging/php-master/includes/libs/objectcache/MemcachedPeclBagOStuff.php on line 61

Although php7.2-memcached seems installed:

[thcipriani@deployment-deploy01 ~]$ apt-cache policy php-memcached
php-memcached:
  Installed: 3.0.1+2.2.0-1~wmf1
  Candidate: 3.0.1+2.2.0-1~wmf1
  Version table:
     3.0.1+2.2.0-1 500
        500 http://deb.debian.org/debian stretch/main amd64 Packages
 *** 3.0.1+2.2.0-1~wmf1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/thirdparty/php72 amd64 Packages
        100 /var/lib/dpkg/status
hashar added a subscriber: Joe.Mar 5 2019, 4:02 PM

beta-scap-eqiad constantly fails now. The last good build was today March 5th at 11:43:02

On deployment-deploy01 scap got upgraded but that does not seem related

Start-Date: 2019-03-05  11:42:54
Commandline: apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install scap=3.9.2-1
Upgrade: scap:amd64 (3.9.1-1, 3.9.2-1)
End-Date: 2019-03-05  11:42:56

Still from /var/log/apt/history.log, some php packages have been removed/installed by @Joe :

Start-Date: 2019-03-01  06:58:43
Commandline: apt-get remove --purge php-memcached
Requested-By: oblivian (4816)
Purge: php-memcached:amd64 (3.0.1+2.2.0-1)
End-Date: 2019-03-01  06:58:45
Start-Date: 2019-03-01  06:59:32
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php-memcached
Requested-By: oblivian (4816)
Install: php-memcached:amd64 (3.0.1+2.2.0-1~wmf1)
End-Date: 2019-03-01  06:59:33
thcipriani triaged this task as High priority.Mar 5 2019, 7:16 PM
thcipriani added a subscriber: MoritzMuehlenhoff.

I have disabled beta-scap-eqiad since it's now failing constantly.

Other clues:

Noticed this this morning, which is strange:

$ strace -ofile -f php7.2 -i
open("/usr/lib/php/20170718/memcached.so", O_RDONLY|O_CLOEXEC) = 4                                                                                    
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)                                                                       
open("/usr/lib/x86_64-linux-gnu/libmemcached.so.11", O_RDONLY|O_CLOEXEC) = 4                                                                          
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)                                                                       
open("/usr/lib/x86_64-linux-gnu/libmemcachedutil.so.2", O_RDONLY|O_CLOEXEC) = 4                                                                       
open("/usr/lib/php/20170718/memcached.so.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
In T217587#5001854, @hashar wrote:

Still from /var/log/apt/history.log, some php packages have been removed/installed by @Joe :

Start-Date: 2019-03-01  06:58:43
Commandline: apt-get remove --purge php-memcached
Requested-By: oblivian (4816)
Purge: php-memcached:amd64 (3.0.1+2.2.0-1)
End-Date: 2019-03-01  06:58:45
Start-Date: 2019-03-01  06:59:32
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php-memcached
Requested-By: oblivian (4816)
Install: php-memcached:amd64 (3.0.1+2.2.0-1~wmf1)
End-Date: 2019-03-01  06:59:33

This may be related to a change by @MoritzMuehlenhoff from a few hours ago: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494212/

Stashbot subscribed.Mar 5 2019, 7:17 PM

Mentioned in SAL (#wikimedia-releng) [2019-03-05T19:17:26Z] <thcipriani> disable beta-scap-eqiad due to T217587

thcipriani renamed this task from scap in job beta-scap-eqiad has random failures: Call to mwscript eval.php returned: None to Class 'Memcached' not found for php7 in beta.Mar 5 2019, 7:19 PM
Joe added a comment.Mar 5 2019, 8:06 PM

apt-get install php7.2-msgpack solved the problem FWIW.

Joe closed this task as Resolved.Mar 5 2019, 8:06 PM
Joe claimed this task.
hashar added a comment.Mar 6 2019, 8:23 AM

And I think the php package fixed eventually caused another issue with curl when running update.php T217733

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits