Page MenuHomePhabricator
Create Task
Maniphest T208756

New Cloud VPS project 'cloud-analytics'
Closed, ResolvedPublic
Actions

Description

In T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster we determined that we will be setting up a Hadoop & Presto cluster in Cloud VPS under a new project cloud-analytics. We'll be adding some new hardware to Cloud VPS networks for worker nodes, but we also need some virt resources for various master and coordination nodes.

We need 3 more VMs for HA Hadoop masters and Hive etc. 'coordinator' nodes. These should be:
Each should have 16G RAM and 8 cores. (If this is too highly speced, we can be flexible here).

Related Objects

Event Timeline

Ottomata created this task.Nov 5 2018, 6:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 5 2018, 6:02 PM
Ottomata mentioned this in T207205: Set up 3 Ganeti VMs for datalake cloud analytics Hadoop cluster.Nov 5 2018, 6:02 PM
Ottomata updated the task description. (Show Details)
Ottomata mentioned this in T207321: Figure out networking details for new cloud-analytics-eqiad Hadoop/Presto cluster.
• GTirloni assigned this task to Andrew.Nov 6 2018, 4:34 PM
• GTirloni triaged this task as Medium priority.
• GTirloni added a project: cloud-services-team (Kanban).
• bd808 moved this task from Inbox to Clinic Duty on the cloud-services-team (Kanban) board.Nov 7 2018, 1:10 AM
• bd808 moved this task from Inbox to Approved on the Cloud-VPS (Project-requests) board.Nov 13 2018, 4:30 PM
Andrew closed this task as Resolved.Nov 14 2018, 10:13 PM

I've created this project -- of course we'll be waiting for the hardware to get racked before it can do much.

Regarding the usage of the support nodes: you currently have default new-project quotas, which are 8 instances, 8 cores, 16Gb RAM. I'd encourage you to do some testing with the stock VM sizes that are available; if you turn out to be maxing out your existing specs then we can adjust your quotas as needed.

(And, of course, your quotas will have to be increased immensely to permit scheduling on the dedicated hardware. I'll cross that bridge when I come to it.)

In the meantime, @Ottomata is the project admin of this project. You can add other team members as you see fit via Horizon.

Ottomata added a comment.Nov 15 2018, 2:32 PM

PERFECT, thanks Andrew! We'll try to fit what we need in the stock VMs first. Sounds good.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits