Note, it seems searching for files on Commons is currently impossible. I've created T295480 for that.

Notes from irc:

• commonwiki_file is empty and has no alias
• moving traffic to the backup cluster

After doing some testing, I have a rough recovery plan:

1. Deploy elasticsearch-repository-swift plugin to eqiad and codfw clusters
2. Configure both clusters to connect to ms-fe.svc.eqiad.wmnet (swift)
3. Snapshot the existing commonswiki_file index from the codfw cluster to swift, take note of start time
4. Restore the snapshot from swift to the eqiad cluster.
5. Run CirrusSearch downtime catchup procedure against eqiad for the period between starting restore and the cluster no longer failing writes to the commonswiki index.

Some related notes:

• elasticsearch-repository-swift was never released for 6.5.4, I ended up taking the last commit targeting 6.6.0 and compiling it against 6.5.4 (change elasticsearchVersion = 6.5.4, and change gradle from 5 to 4.1). What process should we follow to include this in the plugins .deb since we are no longer the upstream here?
• Should we have a separate auth setup in swift for cirrussearch snapshots?
• By default snapshot backup/restore is limited to 20MB/s per partition. Since commonswiki is 32 partitions the cluster will limit itself to 640MB/s, or over 5 gigabits/s. I suspect this is a bit excessive for the swift cluster, or at least beyond doubling the typical network traffic. What would a more appropriate limit be? @fgiunchedi
• After or during restore of the snapshot we likely need to manually assign the commonswiki_file and commonswiki aliases to it.

Closing this ticket, as this specific problem has been resolved. New ticket for remaining work: T295705

Mentioned in SAL (#wikimedia-operations) [2021-11-23T17:35:41Z] <ebernhardson> T295478 start snapshot of commonswiki_file from cirrus codfw -> swift eqiad