That's somewhat expected, at this point nutcracker is not enabled for automatic service startup:

jmm@mw1293:~$ sudo systemctl is-enabled nutcracker
disabled

And the puppet module then starts it (from modules/nutcracker/manifests/init.pp):

service { 'nutcracker':
    ensure => ensure_service($ensure),
}

I'm not sure if that's intended, it might simply be a side effect of the trusty -> jessie migration. I fortunately forgot most I ever knew about Upstart, but nutcracker probably had an automatic service startup under trusty? Adding @Joe and @ori who probably know the history best.

Convering nutcracker to base::service_unit is likely the best fix here.

Also, since HHVM in our current config unconditionally tries to connect to unix::/var/run/nutcracker/redis_$DC.sock would could also add a "After=nutcracker.service" to HHMV's unit.

In T163795#3254246, @MoritzMuehlenhoff wrote:

Also, since HHVM in our current config unconditionally tries to connect to unix::/var/run/nutcracker/redis_$DC.sock would could also add a "After=nutcracker.service" to HHMV's unit.

Sounds like the right thing to do

I dug a little deeper and this turned out to be a subtle packaging bug / debhelper oddity:

The current debian/rules file uses:

dh $@  --with autoreconf --with-systemd

Looks totally benign, right? Here's the postinst which gets generated by dh from that:

# Automatically added by dh_installinit
# In case this system is running systemd, we need to ensure that all
# necessary tmpfiles (if any) are created before starting.
if [ -d /run/systemd/system ] ; then
        systemd-tmpfiles --create /usr/lib/tmpfiles.d/nutcracker.conf >/dev/null || true
fi
# End automatically added section
# Automatically added by dh_installinit
if [ -x "/etc/init.d/nutcracker" ]; then
        update-rc.d nutcracker defaults >/dev/null
        invoke-rc.d nutcracker start || exit $?
fi
# End automatically added section

But if we change the dh call to this:

dh $@  --with autoreconf,systemd

Then after a rebuild, debhelper inserts the correct code to activate and start nutcracker:

# Automatically added by dh_systemd_enable
# This will only remove masks created by d-s-h on package removal.
deb-systemd-helper unmask nutcracker.service >/dev/null || true

# was-enabled defaults to true, so new installations run enable.
if deb-systemd-helper --quiet was-enabled nutcracker.service; then
        # Enables the unit on first installation, creates new
        # symlinks on upgrades if the unit file has changed.
        deb-systemd-helper enable nutcracker.service >/dev/null || true
else
        # Update the statefile to add new symlinks (if any), which need to be
        # cleaned up on purge. Also remove old symlinks.
        deb-systemd-helper update-state nutcracker.service >/dev/null || true
fi
# End automatically added section
# Automatically added by dh_installinit
# In case this system is running systemd, we need to ensure that all
# necessary tmpfiles (if any) are created before starting.
if [ -d /run/systemd/system ] ; then
        systemd-tmpfiles --create /usr/lib/tmpfiles.d/nutcracker.conf >/dev/null || true
fi
# End automatically added section
# Automatically added by dh_installinit
if [ -x "/etc/init.d/nutcracker" ]; then
        update-rc.d nutcracker defaults >/dev/null
        invoke-rc.d nutcracker start || exit $?
fi
# End automatically added section

I'll upload a fixed package to jessie-wikimedia after some more tests in codfw.

The new package fixes that (tested by rebooting two servers with and without the new package:)

root@mw2222:~# dpkg -l nutcracker
ii nutcracker 0.4.1-1+wm3~jessie0 amd64 Fast, light-weight proxy for memcached and Redis
root@mw2222:~# grep "Failed connecting to redis server" /var/log/hhvm/error.log | wc -l
26

root@mw2220:~# dpkg -l nutcracker
ii nutcracker 0.4.1-1+wm3~jessie1 amd64 Fast, light-weight proxy for memcached and Redis
root@mw2220:~# grep "Failed connecting to redis server" /var/log/hhvm/error.log | wc -l
0

Mentioned in SAL (#wikimedia-operations) [2017-05-19T11:05:09Z] <moritzm> uploaded nutcracker 0.4.1-1+wm3~jessie1 to apt.wikimedia.org (T163795)

stretch now has 0.4.1 (prepared/maintained by yours truly) and I just checked, doesn't suffer from this bug. The right way here would be for us to switch to that, or if we have local patches, rebase them on top of stretch's rather than keep stacking patches on top of our fork.

I've filed T166038 for rebasing to the stretch 0.4.1 package. I'll proceed with rolling out the current isolated fix in the mean time; with the current behaviour nutcracker is tripping Icinga alerts if more than one mw* server is rebooted at a time.

Mentioned in SAL (#wikimedia-operations) [2017-05-23T14:03:50Z] <moritzm> installing nutcracker update in codfw (T163795)