Page MenuHomePhabricator

refresh hardware for logstash100[123]
Closed, DeclinedPublic

Description

logstash100[123] are getting old and need replacement. Those servers are used only as logstash endpoints and are thus not consuming much in term of IO.

Current specs:

  • Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz
  • 6 CPU, not hyperthreaded
  • 2x 500GB spinning HDD (only 50GB allocated)
  • 16 GB RAM

Current usage (over the last 90 days):

  • CPU peaks at ~10% utilisation
  • IOPS are close to zero (peak of 2 write IOPS)
  • network peaks at 10MB/s, 300 UDP packets/s, 200 established TCP connections
  • including caches, ~7.5GB of RAM used.

@bd808 suggested that we could move those machines to VMs on Ganeti, but I'm not entirely sure what that implies.

New specs (on real H/W):
Since @RobH is trying to standardize our server specs, this should probably go to the smallest configuration we have.

New specs (VM, spec per VM / total):

  • CPU: 4 / 12
  • HDD: 50GB / 150GB
  • RAM: 8GB / 24GB

note: it looks like the Ganeti cluster has enough resources to accommodate those new VMs.

Event Timeline

Gehel created this task.Aug 14 2017, 9:27 AM
RobH added a subscriber: akosiaris.Aug 17 2017, 4:36 PM

This does indeed look like it could go into a gaeneti VM, since they are very low requirements.

"@bd808 suggested that we could move those machines to VMs on Ganeti, but I'm not entirely sure what that implies."

Basically we toss it into vm-requests not hardware-requests and @akosiaris reviews that project. Once reviewed and approved, either Alex, Daniel, or myself can make the VM and get it spun up. I'd like to do so, since I've not created one since Alex re-balanced the VM's between rows, but any root can make it.

The full details for requests on VMs is here: https://wikitech.wikimedia.org/wiki/Operations_requests#Virtual_machine_requests_.28Production.29

With the small requirements of these hosts, putting them into a ganeti VM does seem quite feasible. It also is cheaper, since shared hw cost in the VM cluster is cheaper for small service requirements like this than bare hardware.

Gehel reassigned this task from Gehel to akosiaris.Aug 17 2017, 4:57 PM

@akosiaris: could you have a look into this request and let me know if it make sense to move the logstash ingestion nodes to ganeti?

@Gehel, yes it does. In fact it's a pretty good idea. Resource wise, these hosts look like perfect candidates (the mem requirement of 8GB is a tad high but OK). Fire up the vm-requests and will get them done in no time.

There is one small point. I see logstash100[1-3] are spread across 3 rack rows. This is not currently possible with ganeti as we have it spread across only 2 rows. From what I gather this should not be a problem, let me know otherwise. Make sure to point out in the request which 2 hosts should reside on the same row.