Page MenuHomePhabricator

switch wikitech to PHP 7.2
Closed, ResolvedPublic

Description

Switch wikitech.wikimedia.org to PHP 7.2

Starting with this change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/509916

Event Timeline

Dzahn created this task.May 15 2019, 5:23 PM
Dzahn added a comment.May 15 2019, 5:52 PM

We now have both PHP 7.0 and PHP 7.2 packages installed but we are still using HHVM.

https://wikitech.wikimedia.org/w/info.php

Change 509926 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labweb/wikitech: start using PHP-FPM

https://gerrit.wikimedia.org/r/509926

Change 509926 merged by Andrew Bogott:
[operations/puppet@production] labweb/wikitech: enable PHP-FPM

https://gerrit.wikimedia.org/r/509926

Change 510949 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikitech: change Apache config from hhvm to php-fpm

https://gerrit.wikimedia.org/r/510949

jbond added a subscriber: jbond.May 29 2019, 10:54 AM
bd808 added a comment.May 29 2019, 5:23 PM
$ ssh labweb1001.wikimedia.org
$ sql labswiki
Fatal error: Uncaught RuntimeException: RedisConnectionPool requires a Redis client library. See https://www.mediawiki.org/wiki/Redis#Setup in /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/redis/RedisConnectionPool.php:80
Stack trace:
#0 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/redis/RedisConnectionPool.php(152): RedisConnectionPool->__construct(Array, '4b1773007d25335...')
#1 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/objectcache/RedisBagOStuff.php(77): RedisConnectionPool::singleton(Array)
#2 /srv/mediawiki/php-1.34.0-wmf.7/includes/objectcache/ObjectCache.php(216): RedisBagOStuff->__construct(Array)
#3 /srv/mediawiki/php-1.34.0-wmf.7/includes/ServiceWiring.php(281): ObjectCache::newFromParams(Array)
#4 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/services/ServiceContainer.php(430): Wikimedia\Services\ServiceContainer->{closure}(Object(MediaWiki\MediaWikiServices))
#5 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/services/ServiceContainer.php(414): Wikimedia\Services\ServiceContainer->createService('MainObjectS in /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/redis/RedisConnectionPool.php on line 80

I'm going to poke around in puppet a bit and try to figure out what manifest we are missing.

bd808 added a comment.May 29 2019, 5:46 PM

I'm going to poke around in puppet a bit and try to figure out what manifest we are missing.

It seems that we have all the correct manifests, but that some of the php packages are out of date following the switch to using the php7.2 component. The deb for php-redis was version 3.1.1-1 on labweb1001. apt-get install php-redis brought in version 4.2.0-1+0~20190222131115.11+stretch~1.gbpf8a642+wmf1 which has Provides: php7.2-redis in its description. php7.2 -m is still not showing the module as loading however. I do see a /etc/php/7.2/cli/conf.d/20-redis.ini config file, but these seem to not all be loading even though php --ini says they are parsed?

$ php7.2 --ini
Configuration File (php.ini) Path: /etc/php/7.2/cli
Loaded Configuration File:         /etc/php/7.2/cli/php.ini
Scan for additional .ini files in: /etc/php/7.2/cli/conf.d
Additional .ini files parsed:      /etc/php/7.2/cli/conf.d/10-mysqlnd.ini,
/etc/php/7.2/cli/conf.d/10-opcache.ini,
/etc/php/7.2/cli/conf.d/10-pdo.ini,
/etc/php/7.2/cli/conf.d/15-xml.ini,
/etc/php/7.2/cli/conf.d/20-apcu.ini,
/etc/php/7.2/cli/conf.d/20-bcmath.ini,
/etc/php/7.2/cli/conf.d/20-bz2.ini,
/etc/php/7.2/cli/conf.d/20-calendar.ini,
/etc/php/7.2/cli/conf.d/20-ctype.ini,
/etc/php/7.2/cli/conf.d/20-curl.ini,
/etc/php/7.2/cli/conf.d/20-dba.ini,
/etc/php/7.2/cli/conf.d/20-dom.ini,
/etc/php/7.2/cli/conf.d/20-excimer.ini,
/etc/php/7.2/cli/conf.d/20-exif.ini,
/etc/php/7.2/cli/conf.d/20-fileinfo.ini,
/etc/php/7.2/cli/conf.d/20-ftp.ini,
/etc/php/7.2/cli/conf.d/20-gd.ini,
/etc/php/7.2/cli/conf.d/20-geoip.ini,
/etc/php/7.2/cli/conf.d/20-gettext.ini,
/etc/php/7.2/cli/conf.d/20-gmp.ini,
/etc/php/7.2/cli/conf.d/20-iconv.ini,
/etc/php/7.2/cli/conf.d/20-igbinary.ini,
/etc/php/7.2/cli/conf.d/20-intl.ini,
/etc/php/7.2/cli/conf.d/20-json.ini,
/etc/php/7.2/cli/conf.d/20-luasandbox.ini,
/etc/php/7.2/cli/conf.d/20-mbstring.ini,
/etc/php/7.2/cli/conf.d/20-msgpack.ini,
/etc/php/7.2/cli/conf.d/20-mysqli.ini,
/etc/php/7.2/cli/conf.d/20-pdo_mysql.ini,
/etc/php/7.2/cli/conf.d/20-phar.ini,
/etc/php/7.2/cli/conf.d/20-posix.ini,
/etc/php/7.2/cli/conf.d/20-readline.ini,
/etc/php/7.2/cli/conf.d/20-redis.ini,
/etc/php/7.2/cli/conf.d/20-shmop.ini,
/etc/php/7.2/cli/conf.d/20-simplexml.ini,
/etc/php/7.2/cli/conf.d/20-sockets.ini,
/etc/php/7.2/cli/conf.d/20-sysvmsg.ini,
/etc/php/7.2/cli/conf.d/20-sysvsem.ini,
/etc/php/7.2/cli/conf.d/20-sysvshm.ini,
/etc/php/7.2/cli/conf.d/20-tokenizer.ini,
/etc/php/7.2/cli/conf.d/20-wddx.ini,
/etc/php/7.2/cli/conf.d/20-wikidiff2.ini,
/etc/php/7.2/cli/conf.d/20-xmlreader.ini,
/etc/php/7.2/cli/conf.d/20-xmlwriter.ini,
/etc/php/7.2/cli/conf.d/20-xsl.ini,
/etc/php/7.2/cli/conf.d/25-memcached.ini
$ for f in /etc/php/7.2/cli/conf.d/*.ini; do
  m1=${f##*-};
  m=${m1%%.ini};
  (php7.2 -m | grep -q $m)||echo $m;
done
opcache
apcu
geoip
igbinary
luasandbox
msgpack
phar
redis
simplexml
wikidiff2
memcached
Joe added a comment.May 30 2019, 6:30 AM

For the record, the above script works once you do

sudo apt-get install php-apcu php-bcmath php-bz2 php-cli php-common php-curl php-gd php-geoip php-gmp php-igbinary php-intl php-luasandbox php-mail php-mail-mime php-mbstring php-memcached php-mongodb php-msgpack php-mysql php-pear php-redis php-tideways-xhprof php-wikidiff2 php-xml

Fixed on both labwebs.

Joe added a comment.May 30 2019, 6:50 AM

Oh I forgot to add: the list of unloadable extensions could be found in the php7.2-fpm log. You also need to restart php7.2-fpm for it to be effective.

Once you've done that, I'd say let's be bold and just change the proxy/rewrite rules from

fcgi://127.0.0.1:9000

to

proxy:unix:/run/php/fpm-www.sock|fcgi://localhost

everywhere in the configuration, or even use mediawiki::web::vhost.

Also, you probably want to add the profile::mediawiki::php::monitoring to collect metrics and have an admin endpoint available.

Meno25 added a subscriber: Meno25.May 31 2019, 4:28 PM
jijiki triaged this task as Low priority.Jun 21 2019, 8:32 AM
Jdforrester-WMF raised the priority of this task from Low to High.Mon, Oct 7, 9:21 PM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

If this isn't done before tomorrow, the train rollout will break wikitechwiki. :-(

Krenair added a subscriber: Krenair.Mon, Oct 7, 9:24 PM

If this isn't done before tomorrow, the train rollout will break wikitechwiki. :-(

Isn't a train blocker then?

Well, we can pin this wiki for special treatment for a week or two if needed.

bd808 reassigned this task from Dzahn to Andrew.Mon, Oct 7, 11:12 PM

Stealing this cookie from @Dzahn and handing it to @Andrew. I did a little work on the Gerrit patch @Dzahn has started (thank you!), and I think it will work or be close enough to working that we can figure out what is missing. @Bstorm and I decided 2019-10-07 had been enough "fun" during our work hours to put off trying the change until 2019-10-08.

Joe added a comment.Tue, Oct 8, 6:18 AM

If this isn't done before tomorrow, the train rollout will break wikitechwiki. :-(

So we did break HHVM compatibility in the train. I would've preferred that we (SRE) had a final say on when we would be ready to kill it definitively. For instance, there are still some tools that deployers are supposed to be using that have not been converted (fatalmonitor) - we would have prioritized moving wikitech and such small tools beforehand. Also, we still have hhvm installed everywhere and still running and still usable on the mwmaint servers.

But more importantly, I am personally still not 100% convinced an emergency rollback was completely excluded. And I would've liked to have a couple more weeks where we had that bailout plan if something went spectacularly wrong.

This is not a huge problem at the time of writing, but we definitely need better coordination in the future. The gist of this is: before *actually* dropping support for something like an interpreter version, please sync with SRE, and maybe ask for our +1 on the patches that break compatibility.

bd808 moved this task from Backlog to Config on the wikitech.wikimedia.org board.Tue, Oct 8, 9:31 PM
bd808 added a comment.Wed, Oct 9, 7:12 PM

Spotted while using eval.php on labweb1002: we are currently missing the php7.2-ldap package there.

Change 541901 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[operations/mediawiki-config@master] Rollback labswiki to 1.34.0-wmf.25

https://gerrit.wikimedia.org/r/541901

Change 541901 merged by jenkins-bot:
[operations/mediawiki-config@master] Rollback labswiki to 1.34.0-wmf.25

https://gerrit.wikimedia.org/r/541901

bd808 added a comment.Wed, Oct 9, 7:23 PM

Spotted while using eval.php on labweb1002: we are currently missing the php7.2-ldap package there.

Correction, the package appears to be installed, but not activated in /etc/php/7.2/**/conf.d.

Change 510949 merged by Andrew Bogott:
[operations/puppet@production] wikitech: switch runtime from HHVM to PHP 7.2

https://gerrit.wikimedia.org/r/510949

Dzahn closed this task as Resolved.EditedWed, Oct 9, 11:39 PM

php-fpm was already running. Apache config switched over by @Andrew and @bd808 today.