Page MenuHomePhabricator

Varnish reload failing on deployment-cache-upload06
Closed, ResolvedPublic

Description

Noticed that puppet is failing on deployment-cache-upload06:

Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Command failed with error code 106
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Message from VCC-compiler:
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Incompatible VMOD std
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: 	File name: /usr/lib/x86_64-linux-gnu/varnish/vmods/libvmod_std.so
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: 	ABI mismatch, expected <Varnish 6.0.7 525d371e3ea0e0c38edd7baf0f80dc226560f26e>, got <Varnish 6.0.8 97e54ada6ac578af332e52b44d2038bb4fa4cd4a>
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: ('/etc/varnish/wikimedia_upload-frontend.vcl' Line 6 Pos 8)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: import std;
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: -------###-
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Running VCC-compiler failed, exited with 2
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: VCL compilation failed
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-02fa1613-64af-4b28-b3fb-4cee1d7a475d /etc/varnish/wikimedia_upload-frontend.vcl"
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 179, in <module>
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     main()
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 162, in main
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     main_vcl_id = load(vadm_cmd, args.vcl_file)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 123, in load
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     do_cmd(vcl_load_cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/local/sbin/reload-vcl", line 63, in do_cmd
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     subprocess.check_call(cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:   File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns:     raise CalledProcessError(retcode, cmd)
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.load', 'vcl-02fa1613-64af-4b28-b3fb-4cee1d7a475d', '/etc/varnish/wikimedia_upload-frontend.vcl']' returned non-zero exit status 1.
Error: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_upload-frontend.vcl -d 2 -a && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[upload-frontend]/Exec[retry-load-new-vcl-file-frontend]/returns: change from 'notrun' to ['0'] failed: '/usr/local/sbin/reload-vcl -n frontend -f /etc/varnish/wikimedia_upload-frontend.vcl -d 2 -a && (rm /var/tmp/reload-vcl-failed-frontend; true)' returned 1 instead of one of [0] (corrective)

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2021-10-22T13:42:10Z] <ema> deployment-cache-upload06: restart varnish-frontend, package got upgraded to 6.0.8 T294116

ema claimed this task.

I upgraded varnish to 6.0.8 everywhere (see T292290) and forgot about restarting the service on deployment-cache-upload06. It should be fixed now, thanks @Majavah.