According to @BBlack on IRC:

• Specs is "standard, SSD-based varnish cluster machine configurations".
• we should not need to buy any new hardware, we should be able to repurpose existing machines

Well to be completely clear: should not need to buy any new hardware this quarter - all of them need replacing on standard lifetimes, with the first batch of four coming up in FY16-17 IIRC.

As we already have the hardware, this just needs @mark's approval.

@RobH I'd really appreciate if you could let me do the reclaim / reinstall so that I learn something in the process (this is likely going to take you more time and energy than doing it yourself, but it should pay in the long run...).

In T131880#2182880, @Gehel wrote:

@RobH I'd really appreciate if you could let me do the reclaim / reinstall so that I learn something in the process (this is likely going to take you more time and energy than doing it yourself, but it should pay in the long run...).

@Gehel: Absolutely, It is perfect document testing conditions! We've outlined all the steps on https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Reclaim_or_Decommission, so those should have everything you need to reclaim. Let me know when you start working on these (once Mark approves) and I can shadow and double check them (plus work with you to update the docs if they are missing steps.)

Additionally, I'll need to modify the lifecycle about wiping to note that it isn't actually supported on SSDs and their trim support. My understanding is we need to find and use the trim/wipe utilities of the drive vendor to accomplish that.

However, as these were caching machines, and are becoming new caching machines, the wipe is likely unneeded for these anyhow.

There's no real need to reinstall them. I have patches pending to put them into their proper roles, etc.

The patch series starts at: https://gerrit.wikimedia.org/r/#/c/268236/ , but needs manual rebases at this point.

(it's better to look at T109162, that had all the patch links)

@Gehel: Since this isn't going to end up being a reinstall, I'll ping you to do a reinstall on one of the many I do every week!

Let's move forward with repurposing the existing (ex mobile) Varnish servers for maps. :)

With post-switchover work, a weekend coming, and other misc constraints, @Gehel and I planning to actually do the work on Monday.

@Gehel and I made partial progress on this today, to resume tomorrow. Current situation:

1. all 16x new cache_maps machines are puppetized in their new roles, but the non-eqiad ones aren't yet handling frontend (user-facing) traffic. They're also all upgraded to varnish4 (like the old cache_maps boxes were too).
2. in eqiad, the 2x old/beta cache_maps machines are out of service/decommed at all layers, and the 4x new ones there are in service for all frontend (user-facing) traffic.
3. the new eqiad caches are routing requests via cache_maps:eqiad -> cache_maps:codfw -> kartotherian.svc.codfw.wmnet (whereas the old cache_maps:eqiad contacted kartotherian.codfw directly).

What's saved for tomorrow is to go through the steps to pool up geographically-routed frontend requests into the other 3x DCs (codfw, ulsfo, esams).

Mentioned in SAL [2016-04-26T19:54:37Z] <gehel> restarting pybal on lvs4004 to enable new cache configuration for maps (T131880)

Mentioned in SAL [2016-04-26T20:02:31Z] <bblack> restarting pbyal on lvs4002 to enable new cache configuration for maps (T131880)

Mentioned in SAL [2016-04-26T20:30:12Z] <bblack> restarting pbyal on lvs2005 to enable new cache configuration for maps (T131880)

Mentioned in SAL [2016-04-26T20:32:51Z] <bblack> restarting pbyal on lvs2002 to enable new cache configuration for maps (T131880)

Mentioned in SAL [2016-04-26T20:35:56Z] <bblack> restarting pbyal on lvs3004 to enable new cache configuration for maps (T131880)

Mentioned in SAL [2016-04-26T20:38:25Z] <bblack> restarting pybal on lvs3002 to enable new cache configuration for maps (T131880)

Mentioned in SAL [2016-04-26T21:06:43Z] <gehel> activating geodns for new varnish maps servers (T131880)