See T279158 for dc-ops procurement, T281989 for dc-ops racking. This ticket is to track the search team's part: taking the fresh nodes and bringing them properly into service
Step 1: Set up hieradata
[x] allocate between psi/omega, keeping rows as balanced as possible
Step 2: Enable cirrus roles
- after completion of this step, the new hosts should have joined the cirrus elasticsearch clusters
Step 3: Prepare to decom old hosts
[x] set new master configuration - https://phabricator.wikimedia.org/T294805#7473840
[x] set new cluster replication seeds to new masters (note: we didn't realize this was needed till later on, so we actually did this manually as part of step 4)
[x] manually ban from cluster
Step 4: Actually decom hosts
[] remove cirrus role and run decom cookbooks; then open decom tickets for dc-ops