Details needed for reinstall:

• rolling reinstallations (where only one is offline) or batch?
• we discussed in irc the removal of the larger hard disks, as these won't keep the elasticsearch data any longer. Are dual 500GB then sufficient for this? (If so, we'll create a sub-task for the onsite tech (@Cmjohnson) in ops-eqiad to swap the disks out AFTER wiping them. Keep in mind, this wipe will require the host to be depooled for 24 hours. We don't want to put unwiped disks back on a shelf as spare. If the 24 hour downtime isn't acceptable, we can discuss possibly moving these disks into a spare box for wipe, but that is labor intensive for on-site, and non-preferred.
• if its just dual 500GB, we'll use the standard raid1/lvm formatting. (this formatting normally puts a /srv partition for all our larger data items, would this work for you guys for the new roles of these systems?)

When we are ready to do this we should also move 2 of the three boxes to new racks. Today all three are in the same rack behind the same switch which can lead to catastrophic down time rather than service degradation in the face of certain hardware failures.

(preferrably, different rows too for even more protection)

Once I complete T105101: Upgrade Logstash Elasticsearch cluster to 1.6.0 I think we will be ready to start rebuilding logstash100[1-3]. I think this is the list of things we want to do as part of this task:

• T98042: Update Wikimedia apt repo to include debs for Elasticsearch & Logstash on jessie
• Remove the 2 Seagate ST3000DM001-1CH1 3TB storage disks we added to each host so the disks can go into the spare pool
• Relocate the 2 of the 3 boxes to new racks and rows so that the failure of a single switch doesn't take out all logstash event ingestion and kibana frontends
• Reimage hosts using jessie base image
• Apply logstash, kibana and logstash::apifeatureusage Puppet roles
• Profit!

Each of the three hosts is a SPOF for some log event ingestion.

• logstash1001: HHVM, Apache2, CX, SCA
• logstash1002: OCG, Hadoop, IPSEC
• logstash1003: Parsoid

Additionally MediaWiki is configured to randomly select a server from the pool to send log events to for each individual request. This combination means that any downtime will result in some log event loss and the longer each node is down the more events will be lost. Coordinating rotating all of these services to alternate hosts would be possible but techops will need to help determine if that much coordination is necessary or if instead we can deal with some amount of log event loss during the upgrade.

I've added T104035: logstash partman recipe huge root partition while we're at it re imagining servers

It appears that we are now in the steps of relocating two of the three systems into different racks, correct?

We'll have @Cmjohnson then remove the larger capacity disks and relocate two of the three into different racks.

With the above, it seems safe to proceed to take these system offline for these changes. Please advise if that isn't so. (I've also pinged @bd808 in irc to confirm this.)

In T97545#1452296, @RobH wrote:

It appears that we are now in the steps of relocating two of the three systems into different racks, correct?

We'll have @Cmjohnson then remove the larger capacity disks and relocate two of the three into different racks.

With the above, it seems safe to proceed to take these system offline for these changes. Please advise if that isn't so. (I've also pinged @bd808 in irc to confirm this.)

Yes, this should be safe to do at any time. There will be some loss of log event data while the systems are offline but that shouldn't be the end of the world. Logstash100{1,2,3} currently hold none of the actual stored log data and instead only provide log ingestion via Logstash and the Kibana frontend at https://logstash.wikimedia.org/.

If they are reimaged before T98042 is done then we will need to manually copy over and install the debs to get them back online and processing logs:

• http://apt.wikimedia.org/wikimedia/pool/thirdparty/e/elasticsearch/elasticsearch_1.6.0_all.deb
• http://apt.wikimedia.org/wikimedia/pool/thirdparty/l/logstash/logstash_1.4.2-1-2c0f5a1_all.deb
• http://apt.wikimedia.org/wikimedia/pool/universe/l/logstash-contrib/logstash-contrib_1.4.2-1-efd53ef_all.deb

I will be traveling on 2015-07-20 and probably not online until the SF afternoon on 2015-07-21 so it would be great if they were not reimaged before 2015-07-22 unless someone else can be sure to babysit the process of getting them back to work.

The racks are prepped for the move. Switches have been updated
DNS patch is ready to merge https://gerrit.wikimedia.org/r/#/c/226722/

Just need to do the physical removal of disks and move each server

logstash1001 is going to row A4 ge-4/0/13
logstash1003 is going to row D3 ge-3/0/16

Change 227245 had a related patch set uploaded (by Cmjohnson):
Updating dhcp file for logstash1001-1003 to use jessie installer per phab task https://phabricator.wikimedia.org/T97545

https://gerrit.wikimedia.org/r/227245

Change 227245 merged by Cmjohnson:
Updating dhcp file for logstash1001-1003 to use jessie installer per phab task https://phabricator.wikimedia.org/T97545

https://gerrit.wikimedia.org/r/227245

Change 227258 had a related patch set uploaded (by BryanDavis):
logstash: change ip address for logstash1001 and logstash1003

https://gerrit.wikimedia.org/r/227258

Change 227258 merged by jenkins-bot:
logstash: change ip address for logstash1001 and logstash1003

https://gerrit.wikimedia.org/r/227258