Side note: at this point, the need to increase hardware is more for availability than for scalability.

Constraints:

• We want to be able to continue operations in the case where we loose a datacenter, including normal maintenance operations.
• Each data center is serving traffic in a completely isolated manner, users are sent to a specific datacenter based on network proximity. The implication here is that we don't want to rely on the secondary datacenter for the reliability of the main cluster.
• WDQS sometimes need full data reload, where a server being reloaded needs to be taken out of traffic for multiple days. At the moment, our clusters are composed of 2 nodes, which means that during maintenance, we are running on a single node.

Goal:

• Increase the size of both eqiad and codfw WDQS clusters to 3 nodes (adding 1 node to each cluster)

Notes:

• If budget is an issue, adding a node to eqiad should be the priority. That way, we can operate the eqiad cluster comfortably and in case we need to switch to codfw we accept to be in a degraded situation where major maintenance operations might be delayed. This is only a transitory solution. Once we have active/active clusters, codfw will need to be operable just as much as eqiad.
• This request is similar to T139482

Sizing:

Hardware should be similar to the current wdqs200[12] configuration:

CPU: dual Intel(R) Xeon(R) CPU E5-2620 v3
Disk: 800GB raw raided space SSD
RAM: 96GB

number of servers:
eqiad: 1
codfw: 1 (we might be able to do without the codfw server in the short term if that is an issue)

@Deskana I reassign this to you. Let me know if you need more details or if you want me to move forward and open a hardware request for this.

As both wdqs expansion systems have been ordered and have setup tasks, I'm resolving this hardware-request.