Page MenuHomePhabricator

Move wdqs to an LVS service
Closed, ResolvedPublic

Description

Currently wdqs is defined directly in varnish as a pool of randomized backend hostnames. There should be a real service hostname for the internal service like wdqs.svc.eqiad.wmnet, which is defined in LVS with pybal controlling the pooling of the 3x backends, and then varnish's configuration should be updated to use that hostname rather than enumerating the backends directly. This involves a handful of complex puppet changes and LVS new-service deploys are always a bit "special" (requiring careful manual restarts).

Details

Related Gerrit Patches:
operations/puppet : productionwdqs - configure varnish to use LVS service as backend
operations/puppet : productionwdqs - add icinga check for LVS services
operations/puppet : productionwdqs - LVS configuration
operations/dns : masterwdqs LVS DNS entries

Event Timeline

BBlack created this task.Apr 12 2016, 3:32 PM
Restricted Application added projects: Operations, Wikidata, Discovery. · View Herald TranscriptApr 12 2016, 3:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Apr 12 2016, 6:04 PM
Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Apr 21 2016, 1:47 PM

Change 312216 had a related patch set uploaded (by Gehel):
wdqs LVS DNS entries

https://gerrit.wikimedia.org/r/312216

Change 312223 had a related patch set uploaded (by Gehel):
wdqs - LVS configuration

https://gerrit.wikimedia.org/r/312223

Change 312224 had a related patch set uploaded (by Gehel):
wdqs - add icinga check for LVS services

https://gerrit.wikimedia.org/r/312224

Change 312225 had a related patch set uploaded (by Gehel):
wdqs - configure varnish to use LVS service as backend

https://gerrit.wikimedia.org/r/312225

BBlack moved this task from Triage to LoadBalancer on the Traffic board.Sep 30 2016, 2:01 PM

Change 312216 merged by Gehel:
wdqs LVS DNS entries

https://gerrit.wikimedia.org/r/312216

Mentioned in SAL (#wikimedia-operations) [2016-10-05T13:46:18Z] <gehel> deploying new LVS configuration for WDQS service - T132457

Change 312223 merged by Gehel:
wdqs - LVS configuration

https://gerrit.wikimedia.org/r/312223

Mentioned in SAL (#wikimedia-operations) [2016-10-05T14:11:49Z] <gehel> restarting pybal on lvs1006 - T132457

Mentioned in SAL (#wikimedia-operations) [2016-10-05T14:15:00Z] <gehel> restarting pybal on lvs1009 - T132457

Mentioned in SAL (#wikimedia-operations) [2016-10-05T14:17:31Z] <gehel> restarting pybal on lvs1012 - T132457

Mentioned in SAL (#wikimedia-operations) [2016-10-05T14:18:53Z] <gehel> restarting pybal on lvs1003 - T132457

Mentioned in SAL (#wikimedia-operations) [2016-10-05T14:22:27Z] <gehel> restarting pybal on lvs1006 - T132457

Mentioned in SAL (#wikimedia-operations) [2016-10-05T14:27:40Z] <gehel> restarting pybal on lvs1003 - T132457

Change 312224 merged by Gehel:
wdqs - add icinga check for LVS services

https://gerrit.wikimedia.org/r/312224

Change 312225 merged by BBlack:
wdqs - configure varnish to use LVS service as backend

https://gerrit.wikimedia.org/r/312225

Change 315528 had a related patch set uploaded (by BBlack):
cache_misc: removed wdqs probe ref T132457

https://gerrit.wikimedia.org/r/315528

Change 315528 merged by BBlack:
cache_misc: removed wdqs probe ref T132457

https://gerrit.wikimedia.org/r/315528

BBlack closed this task as Resolved.Oct 12 2016, 5:39 PM
BBlack claimed this task.