Page MenuHomePhabricator

Convert misc cluster to 2-layer
Closed, ResolvedPublic

Description

Convert misc varnish cluster from 1layer config to 2layer config. This removes the last of our 1layer-style clusters for standardization, makes it possible to expand the misc cluster into the cache pops in a normal way, and cleans up a few things related to other coming refactoring work...

Event Timeline

BBlack raised the priority of this task from to Low.
BBlack updated the task description. (Show Details)
BBlack added projects: SRE, Traffic.
BBlack added subscribers: Matanya, BBlack, Aklapper.

Change 255108 had a related patch set uploaded (by BBlack):
misc-cluster 2layer refactor, step 1/3

https://gerrit.wikimedia.org/r/255108

Change 255109 had a related patch set uploaded (by BBlack):
misc-cluster 2layer refactor, step 2/3

https://gerrit.wikimedia.org/r/255109

Change 255110 had a related patch set uploaded (by BBlack):
misc-cluster 2layer refactor, step 3/3

https://gerrit.wikimedia.org/r/255110

Change 255108 merged by BBlack:
misc-cluster 2layer refactor, step 1/3

https://gerrit.wikimedia.org/r/255108

Change 255109 merged by BBlack:
misc-cluster 2layer refactor, step 2/3

https://gerrit.wikimedia.org/r/255109

Change 255110 merged by BBlack:
misc-cluster 2layer refactor, step 3/3

https://gerrit.wikimedia.org/r/255110

The misc cluster is now in a 2-layer multi-DC configuration (including ipsec), but with a few missing pieces to go for full functionality:

  1. Defining dynamic_directors stuff for depooling varnish backends (right now it's static)
  2. Setting up new IPs, GeoDNS, and LVS stuff to actually route users to the other datacenters for termination

Change 255375 had a related patch set uploaded (by BBlack):
cache_misc - Add full set of services to conftool data

https://gerrit.wikimedia.org/r/255375

Change 255376 had a related patch set uploaded (by BBlack):
cache_misc - define global IPs in LVS config data

https://gerrit.wikimedia.org/r/255376

Change 255377 had a related patch set uploaded (by BBlack):
cache_misc - set up LVS services at all DCs

https://gerrit.wikimedia.org/r/255377

Change 255378 had a related patch set uploaded (by BBlack):
cache_misc - switch to conftool dynamic directors

https://gerrit.wikimedia.org/r/255378

Change 255375 merged by BBlack:
cache_misc - Add full set of services to conftool data

https://gerrit.wikimedia.org/r/255375

Change 255376 merged by BBlack:
cache_misc - define global IPs in LVS config data

https://gerrit.wikimedia.org/r/255376

Change 255377 merged by BBlack:
cache_misc - set up LVS services at all DCs

https://gerrit.wikimedia.org/r/255377

Change 255383 had a related patch set uploaded (by BBlack):
cache_misc: add LVS service IPs to balancers themselves

https://gerrit.wikimedia.org/r/255383

Change 255383 merged by BBlack:
cache_misc: add LVS service IPs to balancers themselves

https://gerrit.wikimedia.org/r/255383

Change 255384 had a related patch set uploaded (by BBlack):
cache_misc: fix various icinga monitoring

https://gerrit.wikimedia.org/r/255384

Change 255384 merged by BBlack:
cache_misc: fix various icinga monitoring

https://gerrit.wikimedia.org/r/255384

Change 255378 merged by BBlack:
cache_misc - switch to conftool dynamic directors

https://gerrit.wikimedia.org/r/255378

Ok at this point everything is ready and configured correctly, except the final switch hasn't been flipped to send users to the geographic endpoints, which is: https://gerrit.wikimedia.org/r/#/c/255331/

BBlack claimed this task.

Change 256704 had a related patch set uploaded (by BBlack):
cache_misc: move pass-blocks to layer-common code

https://gerrit.wikimedia.org/r/256704

Change 256704 merged by BBlack:
cache_misc: move pass-blocks to layer-common code

https://gerrit.wikimedia.org/r/256704