Page MenuHomePhabricator

Switch codfw caches to tier2, begin pushing some traffic through them to test
Closed, ResolvedPublic

Description

Currently, the codfw cache clusters are configured logically as a tier-1 site in parallel with eqiad. This is the ideal state we want to be in for the long haul, with both of the tier-1 sites being options for the backend of the tier-2's, and tier-1 cache->app layer routing/failover being a completely separate decision.

However, at present we don't have a good solution for traffic security on the cross-tier-1 cache->app traffic in that scenario (see also: T108580 , T81543#1503321 , T107956 (mostly rejected in favor of the first link at this point, I think)).

So, for now, we should reconfigure codfw to be tier-2 as far as the edge layer is concerned, so that we can start using it for traffic while the other bits are still being sorted out.

Event Timeline

BBlack created this task.Aug 24 2015, 5:02 PM
BBlack raised the priority of this task from to Medium.
BBlack updated the task description. (Show Details)
BBlack added a project: Traffic.
BBlack added a subscriber: BBlack.
Restricted Application added a project: acl*sre-team. · View Herald TranscriptAug 24 2015, 5:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BBlack renamed this task from Switch codfw caches to tier2, being pushing some traffic through them to test to Switch codfw caches to tier2, begin pushing some traffic through them to test.Aug 24 2015, 5:04 PM
BBlack set Security to None.
Restricted Application added a subscriber: Matanya. · View Herald TranscriptAug 24 2015, 5:04 PM
BBlack added a subscriber: faidon.Aug 24 2015, 5:04 PM

Change 233438 had a related patch set uploaded (by BBlack):
Switch codfw to tier2

https://gerrit.wikimedia.org/r/233438

Change 233616 had a related patch set uploaded (by BBlack):
Disable IPSec monitoring temporarily

https://gerrit.wikimedia.org/r/233616

Change 233616 merged by BBlack:
Disable IPSec monitoring temporarily

https://gerrit.wikimedia.org/r/233616

Change 233438 merged by BBlack:
Switch codfw to tier2

https://gerrit.wikimedia.org/r/233438

codfw switch to tier2 is complete. I don't *think* there's any need to wipe caches down there, either. So we're probably ok at this point to move some limited user traffic over. We should perhaps upgrade the LVS to jessie as well first, though (should be trivial, and there's no point having codfw being the only one on trusty at this point).

Jessie LVS upgrades @ codfw successful, and we should be good to go, I think, for e.g. things like: https://gerrit.wikimedia.org/r/#/c/231772

BBlack closed this task as Resolved.Sep 11 2015, 4:36 PM
BBlack claimed this task.

We're pushing Mexico and several US states' traffic through codfw at this point. There's a little more to do in T110530 before we raise the traffic level dramatically, but we're definitely past the "test that basic prod traffic works" phase now.

BBlack moved this task from Triage to Done on the Traffic board.Sep 22 2015, 1:57 PM