Page MenuHomePhabricator
Create Task
Maniphest T138747

Varnishkafka should auto-reconnect to abandoned VSM
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

When Varnish restarts the SHM log file gets abandoned and Varnishkafka exits gracefully as consequence. It would be nice to implement a "auto-reconnect" behavior like varnishlog:

"""
$ sudo varnishlog -g request
Log abandoned
Log reacquired
"""

We are currently rely on systemd to restart Varnishkafka when it fails or shutsdown.

Useful snippet from VUT:
https://github.com/varnishcache/varnish-cache/blob/4.1/lib/libvarnishtools/vut.c#L366

Steps for this task:

  1. Add the code necessary to implement the re-connect feature to Varnishkafka
  2. Code Review, tests
  3. Debian packaging
  4. Deploy to cp Misc and Maps
  5. Deploy to Upload and Text

Details

SubjectRepoBranchLines +/-
Improve resilience during varnish (re)startsoperations/software/varnish/varnishkafkamaster+78 -33
Improve resilience during varnish (re)startsoperations/software/varnish/varnishkafkamaster+78 -33
Customize query in gerrit

Related Objects

Event Timeline

elukey created this task.Jun 27 2016, 9:29 AM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJun 27 2016, 9:29 AM
elukey updated the task description. (Show Details)Jun 27 2016, 9:33 AM
• ema triaged this task as Medium priority.Jun 27 2016, 11:00 AM
• Nuria subscribed.Jul 7 2016, 5:07 PM

What about sequence numbers?

Now when varnish restarts we are restarting varnishkafka and thus sequence numbers go to zero. If varnishkafka reconnects to teh new varnish instance automatically (without a res-tart) sequence numbers will continue increase.

Milimetric moved this task from Incoming to Dashiki on the Analytics board.Jul 7 2016, 5:09 PM
Milimetric moved this task from Dashiki to Backlog (Later) on the Analytics board.
elukey lowered the priority of this task from Medium to Low.Jul 18 2016, 8:41 AM
gerritbot subscribed.Sep 19 2016, 1:29 PM

Change 311415 had a related patch set uploaded (by Elukey):
Improve resilience during varnish restarts

https://gerrit.wikimedia.org/r/311415

gerritbot added a project: Patch-For-Review.Sep 19 2016, 1:29 PM
gerritbot added a comment.Sep 21 2016, 9:39 AM

Change 311415 merged by Elukey:
Improve resilience during varnish (re)starts

https://gerrit.wikimedia.org/r/311415

gerritbot added a comment.Sep 21 2016, 11:23 AM

Change 311965 had a related patch set uploaded (by Elukey):
Improve resilience during varnish (re)starts

https://gerrit.wikimedia.org/r/311965

gerritbot added a comment.Sep 21 2016, 1:07 PM

Change 311965 merged by Elukey:
Improve resilience during varnish (re)starts

https://gerrit.wikimedia.org/r/311965

Stashbot subscribed.Sep 21 2016, 5:40 PM

Mentioned in SAL (#wikimedia-operations) [2016-09-21T17:40:29Z] <elukey> installed varnishkafka 1.0.12-1 on cp3034.esams (T138747)

elukey raised the priority of this task from Low to Medium.Sep 22 2016, 6:09 AM
elukey edited projects, added Analytics-Kanban; removed Patch-For-Review, Analytics.
elukey updated the task description. (Show Details)
elukey set the point value for this task to 8.
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey added a comment.EditedSep 23 2016, 8:54 AM

Rolled out in cache misc / maps / upload (text is still using Varnish 3)

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Sep 23 2016, 8:55 AM
elukey mentioned this in T128374: Sort out analytics service dependency issues for cp* cache hosts.Sep 23 2016, 8:57 AM
• Nuria closed this task as Resolved.Oct 19 2016, 7:23 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL