The goal is to increase the size of both eqiad and codfw clusters to 36 nodes. We want to keep the specs as close as possible to the current server to ensure uniform load.
Specs:
CPU: Dual Intel(R) Xeon(R) CPU E5-2640 v3
Disk: 800GB raw raided space (2x 800GB SSD RAID1 or similar, software RAID is fine)
RAM: 128GB
Number of servers:
eqiad: 36-31 = 5
codfw: 36-24 = 12
numbers are: [desired size of cluster] - [current size of cluster] = [number of servers to add]
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T149088 Increase elasticsearch cirrus clusters to 36 nodes | |||
| Resolved | RobH | T149089 elasticsearch new servers (5x eqiad / 12x codfw) | |||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Resolved | Gehel | T154251 rack/setup/install elastic2025-2036 | |||
| Resolved | RobH | T154605 codfw: elastic2025-elastic2036/switch port configuration | |||
| Resolved | Gehel | T155790 rack/setup/install elastic1048-1052 |
Some questions regarding elastic search specifications:
It should be noted that the cpu Intel(R) Xeon(R) CPU E5-2640 v3 is now a late model CPU, with the new one being Intel(R) Xeon(R) CPU E5-2640 v4. The v3 is 2.6GHz with 8 cores, the v4 2.4GHz with 10 cores. This may require an upgrade in memory to maintain the memory to cpu core ratio currently in use on the elastic search cluster.
If these systems cannot be weighted in load, other considerations for balancing may need to be used, as they won't be identical systems.
The Intel® Xeon® Processor E5-4655 v4(30M Cache, 2.50 GHz) has 8 cores, but it is likely more expensive. I'll get it quoted, but plan that it won't really be a viable option.
The current systems have 2 [cpu] * 8 [cores per cpu] * 2 [hyperthreading] = 32 presented cores
128 GB RAM / 32 = 4GB per presented core
To maintain this for the new revision 4 CPU, we'll need to bring the memory up accordingly.
rev4 is 2 * 10 * 2 = 40
4GB * 40 = 160
RAM would only balance out at 192GB, so that means they'll have 4.8GB per core.
If we want to keep these at only 128GB, that lowers the allocation of memory per presented core down to:
128 / 40 = 3.2GB
So the questions are:
Chatted with @Gehel about this, updating this task and assigning to him for feedback. Please assign back to me once feedback is added.
Thanks!
I neglected to ask:
memory to core count isn't a big deal for our elasticsearch cluster, for the most part we are using memory as a giant disk cache. There is a limit of 30.5G of java heap for a single elasticsearch process, so adding more cores will still have to share the same java heap. @Gehel or @dcausse may be able to comment more, but typically it looks like we generally have 10-15G of static heap usage, and the rest of the heap is used to buffer how often we need to do a big GC. Under load we still only do an old GC every 30min to 1hr,
The short answer there is, i don't think more cores will require more memory.
The next question would be if increasing memory on the machines would help keep more hot data in memory, rather than going out to disk. After an update to 35 machines per cluster the average machine will have ~266GB of data. It looks like our busiest servers are pulling about 10MB/s from the disk to handle queries. If the price difference is pretty minimal 192GB wouldn't hurt, but I wouldn't declare it any kind of requirement.
There was also a question or IRC about if more cores would end up needing more disk space. For reference, current cluster disk usage:
| current | upgrade | after upgrade | |
| eqiad used | 9358G | 9358G | |
| eqiad avail | 9317G | 5*800G | 13317G |
| eqiad percent | 51% | 41% | |
| codfw used | 9319GB | 9319GB | |
| codfw avail | 7595G | 12*800G | 17195G |
| codfw percent | 55% | 35% | |
If anything, using the newer 800G disks is giving us way more space than we need (but if i remember correctly, last time the price difference on 800G disks vs. smaller was almost non-existent). No change will be necessary there.
Re-reading this after some sleep, it seems that @EBernhardson nailed all the answers (as always). We can always find ways to increase memory allocated to elasticsearch (running multiple instances on the same server for example), but the increased complexity is probably not worth the trouble. Nothing to add...
All systems for this have been ordered. The codfw systems are in place, and the eqiad systems will be racked and installed via T154251