Page MenuHomePhabricator

Refactor VCL for applayer datacenter-switching
Closed, ResolvedPublic

Description

Make it possible to switch, with e.g. simple hieradata commits, the source datacenter (eqiad or codfw) for application-layer services contacted by varnish backends. The switches should have the granularity to be switched independently per-service-per-cache-datacenter.

Event Timeline

Change 275115 had a related patch set uploaded (by BBlack):
caches: remove backend_scaled_weights

https://gerrit.wikimedia.org/r/275115

Change 275116 had a related patch set uploaded (by BBlack):
wikimedia-common VCL: remove static backend weighting

https://gerrit.wikimedia.org/r/275116

Change 275117 had a related patch set uploaded (by BBlack):
varnish: get rid of backend_options

https://gerrit.wikimedia.org/r/275117

Change 275118 had a related patch set uploaded (by BBlack):
varnish: allow director backends to be single-value again

https://gerrit.wikimedia.org/r/275118

Change 275119 had a related patch set uploaded (by BBlack):
r::c::config: remove has_ganglia

https://gerrit.wikimedia.org/r/275119

Change 275120 had a related patch set uploaded (by BBlack):
r::c::config: remove lvs::configuration include

https://gerrit.wikimedia.org/r/275120

Change 275121 had a related patch set uploaded (by BBlack):
r::c::config: remove parsoid (unused)

https://gerrit.wikimedia.org/r/275121

Change 275122 had a related patch set uploaded (by BBlack):
r::c::config: add restbase @ codfw

https://gerrit.wikimedia.org/r/275122

Change 275123 had a related patch set uploaded (by BBlack):
r::c::config: move to hieradata

https://gerrit.wikimedia.org/r/275123

Change 275124 had a related patch set uploaded (by BBlack):
varnishes: control applayer DC routing from hieradata

https://gerrit.wikimedia.org/r/275124

Status update: The changes uploaded so far to the cache_app_routing topic should make this part basically work. Waiting for Monday to merge, and may finish with a followup patch to support 'split' routes as well (active:active), but we're basically done with the bulk of resolving this task.

Change 275497 had a related patch set uploaded (by BBlack):
WIP: first attempt at cache_app_route() w/ split

https://gerrit.wikimedia.org/r/275497

Change 275115 merged by BBlack:
caches: remove backend_scaled_weights

https://gerrit.wikimedia.org/r/275115

Change 275116 merged by BBlack:
wikimedia-common VCL: remove static backend weighting

https://gerrit.wikimedia.org/r/275116

Change 275117 merged by BBlack:
varnish: get rid of backend_options

https://gerrit.wikimedia.org/r/275117

Change 275118 merged by BBlack:
varnish: allow director backends to be single-value again

https://gerrit.wikimedia.org/r/275118

Change 275119 merged by BBlack:
r::c::config: remove has_ganglia

https://gerrit.wikimedia.org/r/275119

Change 275120 merged by BBlack:
r::c::config: remove lvs::configuration include

https://gerrit.wikimedia.org/r/275120

Change 275121 merged by BBlack:
r::c::config: remove parsoid (unused)

https://gerrit.wikimedia.org/r/275121

Change 275122 merged by BBlack:
r::c::config: add restbase @ codfw

https://gerrit.wikimedia.org/r/275122

Change 275123 merged by BBlack:
r::c::config: move to hieradata

https://gerrit.wikimedia.org/r/275123

Change 275124 merged by BBlack:
varnishes: control applayer DC routing from hieradata

https://gerrit.wikimedia.org/r/275124

Change 275509 had a related patch set uploaded (by BBlack):
wikimedia-common VCL: add back truly-static backend weighting

https://gerrit.wikimedia.org/r/275509

Change 275509 merged by BBlack:
wikimedia-common VCL: add back truly-static backend weighting

https://gerrit.wikimedia.org/r/275509

BBlack claimed this task.

This works now, and is controlled by per-cluster hieradata in the apps section of e.g. hieradata/common/cache/text.yaml, where per-service route attribute can be switched between eqiad and codfw.

Change 275497 merged by BBlack:
cache_app_route(): parser func for cache->app routing

https://gerrit.wikimedia.org/r/275497