This seems to be partially expected. T158176: Build / migrate to HHVM 3.18 says "3.18.2 is running on the mediawiki canaries, but the wider rollout is held back until after the DC switchover", which seems sensible.

The easiest start here would be to upgrade mwdebug1001/1002 to 3.18.2. That seems to make sense when some real appservers are already on it. They should probably always be updated first before other mw servers anyways, right?

On any other day i would probably just do that since they are debug hosts. Though right now might be a bad moment. The "is running on the canaries" should cover mwdebug* though, shouldn't it.

In T163278#3192272, @Dzahn wrote:

On any other day i would probably just do that since they are debug hosts. Though right now might be a bad moment.

Possibly, but it should also be pretty low impact. Not my call though.

The "is running on the canaries" should cover mwdebug* though, shouldn't it.

You would think so (since we use the debug servers as canaries too)

Looking at the situation on naos, it looks like an accidental upgrade via hhvm-dbg

Initial puppet run, install hhvm

Start-Date: 2017-04-14  00:01:24
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hhvm
Install: libboost-system1.55.0:amd64 (1.55.0+dfsg-3, automatic), libunwind8:amd64 (1.1-3.2, automatic), libgflags2:amd64 (2.0-2.1, automatic), libzip2:amd64 (0.11.2-1.2, automatic), libboost-filesystem1.55.0:amd64 (1.55.0+dfsg-3, automatic), libvpx3:amd64 (1.5.0-2~wm1, automatic), libmcrypt4:amd64 (2.5.8-3.3, automatic), libboost-program-options1.55.0:amd64 (1.55.0+dfsg-3, automatic), hhvm:amd64 (3.12.14+dfsg-1+wmf1), libgoogle-glog0:amd64 (0.3.3-2, automatic), libboost-thread1.55.0:amd64 (1.55.0+dfsg-3, automatic), liblz4-1:amd64 (0.0~r131-2~wmf1, automatic), libodbc1:amd64 (2.3.1-3, automatic), libxslt1.1:amd64 (1.1.28-2+deb8u2, automatic), libdouble-conversion1:amd64 (2.0.1-1, automatic), libtbb2:amd64 (4.2~20140122-5, automatic)
End-Date: 2017-04-14  00:01:40

And a little while later puppet also installs hhvm-dbg which upgrades hhvm and related packages

Start-Date: 2017-04-14  00:21:51
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install hhvm-dbg
Install: libboost-context1.55.0:amd64 (1.55.0+dfsg-3, automatic), libre2-1:amd64 (20140304+dfsg-2, automatic), hhvm-dbg:amd64 (3.18.2+dfsg-1+wmf1)
Upgrade: hhvm-tidy:amd64 (0.1.3~jessie1, 0.1.3~jessie2), hhvm-luasandbox:amd64 (2.0.12~jessie1, 2.0.12~jessie2), hhvm:amd64 (3.12.14+dfsg-1+wmf1, 3.18.2+dfsg-1+wmf1), hhvm-wikidiff2:amd64 (1.4.1, 1.4.1+wmf1)
End-Date: 2017-04-14  00:22:29

terbium will be upgraded to jessie as soon as we've switched over, for the record.

The only thing that we need to upgrade (and probably @MoritzMuehlenhoff has already scheduled it) are the mwdebug servers, since the rest is Trusty and I don't believe that we'll do any attempt to upgrade HHMV in there.

The remaining appservers with 3.12.7+dfsg-1+wmf1~trusty1 are videoscalers running Trusty, that we eventually will migrate to Jessie.

So as far as I know the next intermediate step will be to upgrade HHVM to 3.18 after the switchover (if no more surprises will come up), and possibly work on migrating the remaining Trusty host to Jessie.

It's unproblematic to also upgrade mwdebug* to 3.18.2, the only difference is a backported patch which only shows up in production load after 4-5 hours.

The deployment servers were not meant to use HHVM 3.18, that the experimental archive section is enabled there is because the deployment servers are running a backported git as requested by RelEng. We can simply downgrade HHVM on naos to 3.12 for now, all mediawiki servers will be upgraded to 3.18 after the switchover anyway.

I've downgraded hhvm-related packages back to their non-experimental version.

It looks like the root cause is experimental and main components of jessie-wikimedia having the same preference (1002), so in case of a reimage or new machine asking apt to install a package picks it up from experimental when enabled. I've updated T158583 with this issue too