Page MenuHomePhabricator

Varnish not starting on deployment-cache-text06
Closed, ResolvedPublic

Description

Not sure why, but deployment-cache-text06 varnish is not coming up after a host reboot (caused by T286502).

1Jul 12 18:39:36 deployment-cache-text06 systemd[1]: Starting varnish-frontend (Varnish HTTP Accelerator)...
2Jul 12 18:39:36 deployment-cache-text06 varnish-frontend[16028]: <jemalloc>: Invalid conf pair: lg_dirty_mult:8
3Jul 12 18:39:36 deployment-cache-text06 varnish-frontend[16028]: <jemalloc>: Invalid conf pair: lg_chunk:17
4Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16028]: Warnings:
5Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16028]: Change will take effect when VCL script is reloaded
6Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16028]: Debug: Version: varnish-6.0.7 revision 525d371e3ea0e0c38edd7baf0f80dc226560f26e
7Jul 12 18:39:37 deployment-cache-text06 varnishd[16031]: Version: varnish-6.0.7 revision 525d371e3ea0e0c38edd7baf0f80dc226560f26e
8Jul 12 18:39:37 deployment-cache-text06 varnishd[16031]: Platform: Linux,4.19.0-17-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit
9Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16028]: Debug: Platform: Linux,4.19.0-17-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit
10Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: No -T in shared memory
11Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-ea37407f-562b-4ac8-9385-17a301641829 /etc/varnish/wikimedia_misc-frontend.vcl"
12Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: Traceback (most recent call last):
13Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: File "/usr/local/sbin/reload-vcl", line 179, in <module>
14Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: main()
15Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: File "/usr/local/sbin/reload-vcl", line 138, in main
16Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: separate_vcl_ids = [load(vadm_cmd, vcl_file) for vcl_file in args.separate_vcl_files]
17Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: File "/usr/local/sbin/reload-vcl", line 138, in <listcomp>
18Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: separate_vcl_ids = [load(vadm_cmd, vcl_file) for vcl_file in args.separate_vcl_files]
19Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: File "/usr/local/sbin/reload-vcl", line 123, in load
20Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: do_cmd(vcl_load_cmd)
21Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: File "/usr/local/sbin/reload-vcl", line 63, in do_cmd
22Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: subprocess.check_call(cmd)
23Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
24Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: raise CalledProcessError(retcode, cmd)
25Jul 12 18:39:37 deployment-cache-text06 varnish-frontend[16034]: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.load', 'vcl-ea37407f-562b-4ac8-9385-17a301641829', '/etc/varnish/wikimedia_misc-fronte
26Jul 12 18:39:37 deployment-cache-text06 systemd[1]: varnish-frontend.service: Control process exited, code=exited, status=1/FAILURE
27Jul 12 18:39:37 deployment-cache-text06 varnishd[16031]: Manager got SIGTERM
28Jul 12 18:39:37 deployment-cache-text06 varnishd[16031]: manager stopping child
29Jul 12 18:39:37 deployment-cache-text06 varnishd[16031]: manager dies
30Jul 12 18:39:37 deployment-cache-text06 systemd[1]: varnish-frontend.service: Failed with result 'exit-code'.

Event Timeline

Mentioned in SAL (#wikimedia-releng) [2021-07-12T18:43:52Z] <majavah> deployment-cache-text06 varnish not starting, T286506, causing an outage on text traffic on deployment-prep

Mentioned in SAL (#wikimedia-releng) [2021-07-12T18:45:33Z] <majavah> upgrade deployment-cache-text06 to use varnish 6 (with profile::cache::varnish::frontend::packages_component), and run apt upgrade, T286506

taavi claimed this task.

Fixed by upgrading Varnish component on hiera and running apt-get upgrade.