We now have both PHP 7.0 and PHP 7.2 packages installed but we are still using HHVM.

https://wikitech.wikimedia.org/w/info.php

Change 509926 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] labweb/wikitech: start using PHP-FPM

https://gerrit.wikimedia.org/r/509926

oops, duplicate of both T219185 and T218615

Change 509926 merged by Andrew Bogott:
[operations/puppet@production] labweb/wikitech: enable PHP-FPM

https://gerrit.wikimedia.org/r/509926

Change 510949 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] wikitech: change Apache config from hhvm to php-fpm

https://gerrit.wikimedia.org/r/510949

$ ssh labweb1001.wikimedia.org
$ sql labswiki
Fatal error: Uncaught RuntimeException: RedisConnectionPool requires a Redis client library. See https://www.mediawiki.org/wiki/Redis#Setup in /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/redis/RedisConnectionPool.php:80
Stack trace:
#0 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/redis/RedisConnectionPool.php(152): RedisConnectionPool->__construct(Array, '4b1773007d25335...')
#1 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/objectcache/RedisBagOStuff.php(77): RedisConnectionPool::singleton(Array)
#2 /srv/mediawiki/php-1.34.0-wmf.7/includes/objectcache/ObjectCache.php(216): RedisBagOStuff->__construct(Array)
#3 /srv/mediawiki/php-1.34.0-wmf.7/includes/ServiceWiring.php(281): ObjectCache::newFromParams(Array)
#4 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/services/ServiceContainer.php(430): Wikimedia\Services\ServiceContainer->{closure}(Object(MediaWiki\MediaWikiServices))
#5 /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/services/ServiceContainer.php(414): Wikimedia\Services\ServiceContainer->createService('MainObjectS in /srv/mediawiki/php-1.34.0-wmf.7/includes/libs/redis/RedisConnectionPool.php on line 80

I'm going to poke around in puppet a bit and try to figure out what manifest we are missing.

In T223393#5221742, @bd808 wrote:

I'm going to poke around in puppet a bit and try to figure out what manifest we are missing.

It seems that we have all the correct manifests, but that some of the php packages are out of date following the switch to using the php7.2 component. The deb for php-redis was version 3.1.1-1 on labweb1001. apt-get install php-redis brought in version 4.2.0-1+0~20190222131115.11+stretch~1.gbpf8a642+wmf1 which has Provides: php7.2-redis in its description. php7.2 -m is still not showing the module as loading however. I do see a /etc/php/7.2/cli/conf.d/20-redis.ini config file, but these seem to not all be loading even though php --ini says they are parsed?

$ php7.2 --ini
Configuration File (php.ini) Path: /etc/php/7.2/cli
Loaded Configuration File:         /etc/php/7.2/cli/php.ini
Scan for additional .ini files in: /etc/php/7.2/cli/conf.d
Additional .ini files parsed:      /etc/php/7.2/cli/conf.d/10-mysqlnd.ini,
/etc/php/7.2/cli/conf.d/10-opcache.ini,
/etc/php/7.2/cli/conf.d/10-pdo.ini,
/etc/php/7.2/cli/conf.d/15-xml.ini,
/etc/php/7.2/cli/conf.d/20-apcu.ini,
/etc/php/7.2/cli/conf.d/20-bcmath.ini,
/etc/php/7.2/cli/conf.d/20-bz2.ini,
/etc/php/7.2/cli/conf.d/20-calendar.ini,
/etc/php/7.2/cli/conf.d/20-ctype.ini,
/etc/php/7.2/cli/conf.d/20-curl.ini,
/etc/php/7.2/cli/conf.d/20-dba.ini,
/etc/php/7.2/cli/conf.d/20-dom.ini,
/etc/php/7.2/cli/conf.d/20-excimer.ini,
/etc/php/7.2/cli/conf.d/20-exif.ini,
/etc/php/7.2/cli/conf.d/20-fileinfo.ini,
/etc/php/7.2/cli/conf.d/20-ftp.ini,
/etc/php/7.2/cli/conf.d/20-gd.ini,
/etc/php/7.2/cli/conf.d/20-geoip.ini,
/etc/php/7.2/cli/conf.d/20-gettext.ini,
/etc/php/7.2/cli/conf.d/20-gmp.ini,
/etc/php/7.2/cli/conf.d/20-iconv.ini,
/etc/php/7.2/cli/conf.d/20-igbinary.ini,
/etc/php/7.2/cli/conf.d/20-intl.ini,
/etc/php/7.2/cli/conf.d/20-json.ini,
/etc/php/7.2/cli/conf.d/20-luasandbox.ini,
/etc/php/7.2/cli/conf.d/20-mbstring.ini,
/etc/php/7.2/cli/conf.d/20-msgpack.ini,
/etc/php/7.2/cli/conf.d/20-mysqli.ini,
/etc/php/7.2/cli/conf.d/20-pdo_mysql.ini,
/etc/php/7.2/cli/conf.d/20-phar.ini,
/etc/php/7.2/cli/conf.d/20-posix.ini,
/etc/php/7.2/cli/conf.d/20-readline.ini,
/etc/php/7.2/cli/conf.d/20-redis.ini,
/etc/php/7.2/cli/conf.d/20-shmop.ini,
/etc/php/7.2/cli/conf.d/20-simplexml.ini,
/etc/php/7.2/cli/conf.d/20-sockets.ini,
/etc/php/7.2/cli/conf.d/20-sysvmsg.ini,
/etc/php/7.2/cli/conf.d/20-sysvsem.ini,
/etc/php/7.2/cli/conf.d/20-sysvshm.ini,
/etc/php/7.2/cli/conf.d/20-tokenizer.ini,
/etc/php/7.2/cli/conf.d/20-wddx.ini,
/etc/php/7.2/cli/conf.d/20-wikidiff2.ini,
/etc/php/7.2/cli/conf.d/20-xmlreader.ini,
/etc/php/7.2/cli/conf.d/20-xmlwriter.ini,
/etc/php/7.2/cli/conf.d/20-xsl.ini,
/etc/php/7.2/cli/conf.d/25-memcached.ini

$ for f in /etc/php/7.2/cli/conf.d/*.ini; do
  m1=${f##*-};
  m=${m1%%.ini};
  (php7.2 -m | grep -q $m)||echo $m;
done
opcache
apcu
geoip
igbinary
luasandbox
msgpack
phar
redis
simplexml
wikidiff2
memcached

For the record, the above script works once you do

sudo apt-get install php-apcu php-bcmath php-bz2 php-cli php-common php-curl php-gd php-geoip php-gmp php-igbinary php-intl php-luasandbox php-mail php-mail-mime php-mbstring php-memcached php-mongodb php-msgpack php-mysql php-pear php-redis php-tideways-xhprof php-wikidiff2 php-xml

Fixed on both labwebs.

Oh I forgot to add: the list of unloadable extensions could be found in the php7.2-fpm log. You also need to restart php7.2-fpm for it to be effective.

Once you've done that, I'd say let's be bold and just change the proxy/rewrite rules from

fcgi://127.0.0.1:9000

to

proxy:unix:/run/php/fpm-www.sock|fcgi://localhost

everywhere in the configuration, or even use mediawiki::web::vhost.

Also, you probably want to add the profile::mediawiki::php::monitoring to collect metrics and have an admin endpoint available.

In T223393#5223143, @Joe wrote:

Once you've done that, I'd say let's be bold and just change the proxy/rewrite rules from

https://gerrit.wikimedia.org/r/c/operations/puppet/+/510949/2/modules/openstack/templates/wikitech/wikitech-web.wikimedia.org.erb

In T223393#5553766, @Jdforrester-WMF wrote:

If this isn't done before tomorrow, the train rollout will break wikitechwiki. :-(

Isn't a train blocker then?

Well, we can pin this wiki for special treatment for a week or two if needed.

In T223393#5553766, @Jdforrester-WMF wrote:

If this isn't done before tomorrow, the train rollout will break wikitechwiki. :-(

So we did break HHVM compatibility in the train. I would've preferred that we (SRE) had a final say on when we would be ready to kill it definitively. For instance, there are still some tools that deployers are supposed to be using that have not been converted (fatalmonitor) - we would have prioritized moving wikitech and such small tools beforehand. Also, we still have hhvm installed everywhere and still running and still usable on the mwmaint servers.

But more importantly, I am personally still not 100% convinced an emergency rollback was completely excluded. And I would've liked to have a couple more weeks where we had that bailout plan if something went spectacularly wrong.

This is not a huge problem at the time of writing, but we definitely need better coordination in the future. The gist of this is: before *actually* dropping support for something like an interpreter version, please sync with SRE, and maybe ask for our +1 on the patches that break compatibility.

Spotted while using eval.php on labweb1002: we are currently missing the php7.2-ldap package there.

Change 541901 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[operations/mediawiki-config@master] Rollback labswiki to 1.34.0-wmf.25

https://gerrit.wikimedia.org/r/541901

Change 541901 merged by jenkins-bot:
[operations/mediawiki-config@master] Rollback labswiki to 1.34.0-wmf.25

https://gerrit.wikimedia.org/r/541901

In T223393#5560962, @bd808 wrote:

Spotted while using eval.php on labweb1002: we are currently missing the php7.2-ldap package there.

Correction, the package appears to be installed, but not activated in /etc/php/7.2/**/conf.d.

Change 510949 merged by Andrew Bogott:
[operations/puppet@production] wikitech: switch runtime from HHVM to PHP 7.2

https://gerrit.wikimedia.org/r/510949