Presently all eqiad memcached servers reside in the same rack. This is due to legacy reasons that have to do with the fact that mc* servers have 10GbE fiber and the availability of 10GbE switches in our datacenter.
The servers need to be shuffled across the datacenter for maximum resiliency.
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Resolved | fgiunchedi | T88805 rack and setup restbase production cluster in eqiad | ||
| Resolved | • Cmjohnson | T83551 Split memcached in eqiad across multiple racks/rows | ||
| Resolved | • Cmjohnson | T82259 mc1016 mgmt not working | ||
| Resolved | Joe | T89345 Missing memcached logs | ||
| · · · |
-- worth mentioning mc1017 and mc1018 have never been setup. Linking ticket
(RT6351)
Chris Johnson
Operations Engineer
Wikimedia Foundation, Inc
(415) 578-0844
<cmjohnson at wikimedia>
@Cmjohnson, can you propose a plan here to spread out the memcached servers to 3 10G racks or more?
We'll be adding some Varnish (10G) servers soon as well, so let's keep that into account.
Moving servers to other racks/rows will mean a global rebalance of the cluster at the moment, losing a lot of memcached keys in the process.
I will try to find a way to trick nutcracker not to do that.
Until I investigated that, please hold this task.
So this rebalancing is going to block the deployment of 2 of the 6 Restbase systems.
Can we plan to move mc1017-mc1018 into row D? Or at minimum, out of the rack now to take the space for the restbase systems?
I'd advise possibly not wiring mc1017--1018 wherever they rest until we figure out the full rebalancing plan.
This is not blocked by me at all now - my investigation in rebalancing is done.
Whenever mc1017 and mc1018 are moved and reinstalled I can work on moving traffic off of mc1001-1002 (or whichever other machines we want to move first).
I'll reassign the ticket to Chris.
Some caveats:
So there will be some user-facing impact from this maintenance work.
We should aim at moving at least one server today, possibly 2
What I understood from IRC yesterday is:
Redis locking being confined in one availability zone is a problem, though, and that needs to be adjusted in the config at later point; I think that's what you meant above.
@faidon yes I meant that, I'd just like to reduce the number of potential issues while moving the servers.
DNS entries have been made and are sitting in gerrit for review. https://gerrit.wikimedia.org/r/#/c/190358/
Switch ports have been labeled, enabled and set to the correct vlan
The racks have been prepped with cabling and power strips and racktables updated