We will probably want to upgrade by running parallel clusters of ES 6 and 7.
Fully populating the 7 cluster and having updates to it written in parallel before moving read traffic over and then decommissioning the 6.x cluster.
- deploy a new version of ElasticSearch 7. e.g. build new image, make sure to include the wmf specific plugins etc. (this might be Already built by Adam for suite). This may also involve asking for more compute resources and temporarily spending more money
- configure mediawiki to continue reading from ES6 using the compatibility layer but also start writing to ES7 on edits
- schedule long running tasks (k8s jobs?) to fully reindex ES7.
- One complete configure mediawiki to read from ES7 but continue writing to ES6 (if we then discover issues with ES7 we can revert this config change to restore search for users)
- after being confident that it is working; decomisison ES6.
N.B. For documentation on how to correctly configure mediawiki to read and write from the right clusters at the right time see: https://gerrit.wikimedia.org/g/mediawiki/extensions/CirrusSearch/%2B/HEAD/docs/settings.txt