Page MenuHomePhabricator

eqiad: New Hive / Oozie server node in eqiad Analytics VLAN
Closed, ResolvedPublic

Description

In Analytics' HW budget meeting today, we decided to purchase a new server for Hive and Oozie, rather than re-allocating one of our older R720s. The spare R720s will be reserved for Druid experiements and work this or next quarter.

This request is fairly flexible. There isn't a need for lots of space or disk IO, but lots of memory will be useful. @Cmjohnson we have something laying around with 32G (give or take 16G) of RAM, but few disks? If so, we can use that. Otherwise, I'd like to order a node with 32G RAM and 2 drives for redundancy. This will come out of the Analytics HW budget.

Event Timeline

Ottomata raised the priority of this task from to Medium.
Ottomata updated the task description. (Show Details)
Ottomata added subscribers: JAllemandou, Ottomata, elukey and 2 others.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@RobH, we'd like to move forward with this one as quickly as possible. We were going to use an older OOW Dell for this, and were about to move forward with T110090. We have recently re-evaluated and decided to release the OOW nodes when we can. Hence the new node order.

@mark and @Nuria approve? This should come out of remainder analytics/ops budget for FY2015-2016.

We don't have a spare that meets this criteria, so we would have to allcoate a spare that has 4 * 4TB disks, or order a new system.

We have six of the high performance misc systems in stock in eqiad. These are Dual Intel® Xeon® Processor E5- 2623v3@3.00/4core and 32GB of memory.

We can allocate one of those with @mark's approval. These systems were purchased on T120248 (which lists the cost per system.) We could order new for slightly cheaper (since we would reduce the 4 * 4TB requirement) but likely not enough to warrant not using on hand spares.

Assigning to @mark for his review/approval/feedback on if we can use one of the new spare misc systems or if we should generate a new quote for this.

@Ottomata: is there a row/rack requirement you need this system in? (Or need it to NOT be in?)

RobH renamed this task from New Hive / Oozie server node in eqiad Analytics VLAN to eqiad: New Hive / Oozie server node in eqiad Analytics VLAN.Feb 2 2016, 9:17 PM
RobH moved this task from Backlog to Pending Approval on the hardware-requests board.

You know, recently I've been noticing a lot of IO from the mysql db on the host this will replace, so 4 drives might not be so bad. We'll RAID 10 them or something. I think this spare will be fine.

Please note that we may want to use one of the 6 HP systems that will be added to spares on T125842 (reclaiming and replacing restbase1001-1006). One of these restbase systems would have 4 * 500GB SATA disks, which would meet the criteria of this request nicely. However, they have 64GB memory versus the 32GB in the HPM with 4 * 4TB SATA.

I'm listing this as a potential option for @mark's review along with my past option listed on T124945#1991623:

We don't have a spare that meets this criteria, so we would have to allcoate a spare that has 4 * 4TB disks, or order a new system.

We have six of the high performance misc systems in stock in eqiad. These are Dual Intel® Xeon® Processor E5- 2623v3@3.00/4core and 32GB of memory.

We can allocate one of those with @mark's approval. These systems were purchased on T120248 (which lists the cost per system.) We could order new for slightly cheaper (since we would reduce the 4 * 4TB requirement) but likely not enough to warrant not using on hand spares.

Assigning to @mark for his review/approval/feedback on if we can use one of the new spare misc systems or if we should generate a new quote for this.

Cool, those machines sound good.

Old RESTbase machines indeed seem good for these, but won't be available for use for realistically another 2 weeks at least.

@Ottomata: Can you advise if waiting for these is acceptable? The new restbase machines arrive next week, and we'll likely need to give it a week or so for them to migrate to them.

@RobH, we'll wait. I might move these services in the mean time to a spare old Analytics Dell. We won't be in such a time crunch then.

Excellent, so with this info we can let @mark review and approve (since his only listed question was the waiting period until these are available.)

Once approved this task can be assigned back to @RobH for followup.

Bump! I suppose we are still waiting for these nodes though?

restbase1001-1006 are not yet available. Though getting @mark's approval for the allocation of one for this in advance would eliminate future bottlenecks in approvals.

I approve using one of the old rb servers for this, as soon as available. Let's make sure we have disks for them?

Hm, @RobH, we could alternatively use WMF4541 for this, no?

Bump, @RobH, could we use WMF4541 instead of waiting?

Sorry about that @Ottomata, this was assigned to @mark so I missed your question.

System WMF4541 is an older spare pool system with 4 * 4TB, dual Intel® Xeon® Processor E5-2450, and 64GB RAM.

If that works for you, we'll need @mark to approve the allocation of this, rather than awaiting the old restbase system relcaim (T130752).

@mark: Please approve the allocation of WMF4541 as a spare. These sysetms have less memory than the restbase1001-1006, but this request doesn't need all the memory in the restbase1001-1006, the 64 in WMF4541 should suffice.

Please attach relevant approval and assign back to me.

This hardware-requests is granted, and system WMF4541 is now allocated for this use. Task T130840 now exists for system setup and deployment.