When hardware planning last year, Analytics asked for budget to expand the 2 Kafka main clusters.
We planned to increase RAM to 64G (+32G) on each of the existent brokers, and add +2 brokers to each cluster. Analytics would like to put this task in SRE's backlog. The Kafka main clusters should be owned by SRE, so we want to leave it up to them to decide if and when this is needed.
AFAIK, the load on Kafka main is not planned to increase significantly in the near future.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | elukey | T220387 Transition Kafka main ownership from Analytics Engineering to SRE - (2018-2019 Q4 SRE Goal Tracking Task) | |||
| Resolved | herron | T217359 Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. | |||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Resolved | herron | T226274 (Need By: June 30) rack/setup/install kafka-main100[1-5] | |||
| Resolved | None | T225005 Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345] | |||
| Resolved | elukey | T288825 Rebalance kafka partitions in main-{eqiad,codfw} clusters |
According to netbox support for hosts kafka[12]00[123] expired in Dec 2018. After discussing a bit with @Ottomata, a server refresh with higher-spec hardware would be a reasonable course of action to address both server age and capacity.
If folks agree with that approach, the next question that comes to mind is do we have budget to proceed with that as part of Q4 work? Or would this need to wait until the new FY?
In the SRE spreadsheet I can see that the suggested replacement FY is 20/21, not the upcoming one.. Just adding the info, not sure if these servers are eligible or not for refresh before the 5y of usage.
After some discussions off-task we do have permission from SRE management to proceed with a HW refresh into higher spec systems. Which to me leads to the question of what spec will be sufficient for the next 2-3 years?
I'm inclined to err on the side of caution and spec in the ballpark of 128G RAM hosts. But at the same time we have a sub-goal (T220389) this quarter to review current capacity and plan for the next 2-3y. So fleshing those details out seems a good place to start.
128GB sounds good, the more page cache Kafka has the better :)
I'd also think about SSD disks (IIRC we don't have them now), and I'd also think about checking if 10G interfaces are needed or not. The current traffic, even per second one analyzed via ifstat, seems to fit well into 1G bandwidth but not sure about the future. If we are planning to add a lot more consumers, for example, TX bandwidth might get close to warning levels (Jumbo is starting to complain about 1G bandwidth for example - https://phabricator.wikimedia.org/T220700 - but it is of course a different use case).
+1 for SSDs. We will be directly exposing time-based messages to consumers, so I imagine random disk access will greatly increase in the mid-term.
Today we discussed desired hardware configs and expansion strategies during a meeting with @elukey @mobrovac @Ottomata and myself. Here are the outcomes:
Taking the above into account, and considering upgrading existing hosts in place is a non-starter, cluster expansion is largely tied to hardware replacement/refresh.
In terms of timing I can think of a few possible approaches:
To circle back on this, we moved forward with option 2 and are using task T225005 to track the migration effort