Page MenuHomePhabricator

MediaWiki beta varnish is down.
Closed, ResolvedPublic

Event Timeline

Noticed that puppet stopped running on beta's deployment-cache-text06 due to c871b021a44108bed44cd044f5177b075ecb322c

After attempting a fix (live hack on deployment-puppetmaster04 changing:

modules/confd/manifests/init.pp
-    Stdlib::Fqdn     $srv_dns       = $facts['domain'],
+    String           $srv_dns       = $::domain,

I kept seeing Abandoned VSM file (Varnish not running?) /var/lib/varnish/frontend/_.vsm

Decided to try to restart the varnish-frontend service following a puppet run, but it looks like the vcc compiler is failing:

Incompatible VMOD netmapper
        File name: /usr/lib/x86_64-linux-gnu/varnish/vmods/libvmod_netmapper.so
        VMOD version 7.1
        varnishd version 6.0
('/etc/varnish/wikimedia_misc-frontend.vcl' Line 13 Pos 8)
import netmapper;
-------#########-

Running VCC-compiler failed, exited with 2
VCL compilation failed
thcipriani triaged this task as Unbreak Now! priority.Nov 6 2020, 8:27 PM

beta down, setting to UBN

thcipriani claimed this task.

Unbreaking puppet seems to have unleashed some interesting behavior: varnish-modules was removed, and libvmod_netmapper was upgraded to an incompatible version.

 apt-get install libvmod-netmapper=1.7-3
apt-get install varnish-modules

Was enough to get back to just slightly broken.

Noticed that puppet stopped running on beta's deployment-cache-text06 due to c871b021a44108bed44cd044f5177b075ecb322c

After attempting a fix (live hack on deployment-puppetmaster04 changing:

modules/confd/manifests/init.pp
-    Stdlib::Fqdn     $srv_dns       = $facts['domain'],
+    String           $srv_dns       = $::domain,

I'm curious if this change made any difference? as i dont think it should (but there could be something with horizons backed which im not familiar with)

I notice that in horizon you set the value of srv_dns to an empty string. However when with the change above i updated things to pass the value directly from profile::conftool::state unless im miss remembering (or puppet changed things) values passed directly like this will always have precedence over anything set hiera (and horizon in this case) as such i wouldn;t expect the above change to have had any impact.

All that aside what is the desired outcome, ultimatly the only place this variable gets used is in the ExecStart on the init script

Environment="CONFD_DISCOVERY=-srv-domain <%= @srv_dns %> -scheme <%= @scheme %>

so an empty value will render as -srv-domain -schema $https which looks like an invalid configuration but im not familiar enough with confd to know what affect this has. Either way if you let me know what you ultimately want to end up in the initi script i can update puppet to comply