Change Details

Similar to the 1.x -> 2.x upgrade, we need to come up with a plan for how we will do the production migration. Likely this can be very similar to the last major upgrade: # Verify (or add necessary flags) the current codebase and the es5 branch can write to a cluster running the other version of elasticseach # Upgrade codfw to 5.x # Schedule a deployment window for CirrusSearch # Merge es5 branch and ship out to all deployed branches (or do on a monday when only one branch is running) # Update mediawiki-config to send all search queries to codfw cluster # Upgrade eqiad to 5.x # Update mediawiki-config to send all search traffic to eqiad again There will be some sticking points that we may not easily be able to get around, but should look into if there are good ways: * The full text search queries we issue to 2.x will not work on 5.x. This is entirely due to the renaming of the 'fields' key to 'stored_fields' in the query we send. Perhaps we can put together a feature flag that briefly disables this while merge+deploy es5 branch * The completion suggester queries we issue to 2.x will not work on 5.x. This is a bigger problem and is not easily solved. We may have to live with a brief service interruption while deploying the merged es5 branch and pushing out the config to switch read traffic to codfw. * Need to double check that our production mappings all load into es5 correctly. Mostly this is the string -> text field type rename, but there might be other issues. ** titlesuggest may have drastically changed, need to test how this works.

Similar to the 1.x -> 2.x upgrade, we need to come up with a plan for how we will do the production migration. Likely this can be very similar to the last major upgrade: # Verify (or add necessary flags) the current codebase and the es5 branch can write to a cluster running the other version of elasticseach # Upgrade codfw to 5.x (hopefully one day activity, one a monday) ## Ship patch to stop sending writes to codfw. Note the time (SAL log or whatever) ## Shut down all hosts in codfw. ## Ship puppet patch to update elasticsearch.yml for 5.x ## Install new elasticsearch.deb ## Bring up the master capable nodes, let them elect a master ## Bring up the rest of the nodes, wait for green ## Reindex the timespan codfw was down. May need some hackishness to let the 2.x maint script talk to 5.x (just the version check) ## Somehow we need to build the 5.x completion suggester indices before the # Update mediawiki-config to send all search queries for next train deployment to codfw cluster. See https://gerrit.wikimedia.org/r/291257 for how we did this last time. Don't forget about TTMServer # Merge es5 branch into master and let it ride the train # After train has rolled forward to all wikis perform the same upgrade actions we did to codfw to eqiad. This will require more coordination to ensure other use cases (api feature logs and phabricator) continue to work. ## Phabricator should be able to point to codfw and do a rebuild from a maint script, then point the search at codfw, then do yet another rebuild to cover any updates that occured between the first rebuild and the switch over. ## Check with @Anomie about api feature log. Perhaps downtime is acceptable here? # Revert prior patch sending all traffic to codfw, allowing traffic to flow to eqiad again. There will be some sticking points that we may not easily be able to get around, but should look into if there are good ways: * The completion suggester indices built by 2.x will sort-of work against 5.x, but not in an ideal manner. We may need some hackery in the code to detect which version of the titlesuggest index is being used and do different things with the results.

Similar to the 1.x -> 2.x upgrade, we need to come up with a plan for how we will do the production migration. Likely this can be very similar to the last major upgrade: # Verify (or add necessary flags) the current codebase and the es5 branch can write to a cluster running the other version of elasticseach # Upgrade codfw to 5.x (hopefully one day activity, one a monday) # Schedule a deployment window for CirrusSearch## Ship patch to stop sending writes to codfw. Note the time (SAL log or whatever) # Merge es5 branch and ship out to all deployed branches (or do on a monday when only one branch is running)## Shut down all hosts in codfw. # Update mediawiki-config## Ship puppet patch to send all search queries to codfw clusterupdate elasticsearch.yml for 5.x # Upgrade eqiad to 5.x## Install new elasticsearch.deb ## Bring up the master capable nodes, let them elect a master ## Bring up the rest of the nodes, wait for green ## Reindex the timespan codfw was down. May need some hackishness to let the 2.x maint script talk to 5.x (just the version check) ## Somehow we need to build the 5.x completion suggester indices before the # Update mediawiki-config to send all search traffic to eqiad again There will be some sticking points that we may not easily be able to get around,queries for next train deployment to codfw cluster. See https://gerrit.wikimedia.org/r/291257 for how we did this last time. but should lookDon't forget about TTMServer # Merge es5 branch into if there are good ways:master and let it ride the train * The full text search queries we issue to 2.x will not work on 5.x. This is entirely due to the renaming of the 'fields' key to 'stored_fields' in the query we sen# After train has rolled forward to all wikis perform the same upgrade actions we did to codfw to eqiad. Perhaps we can put together a featThis will require more coordination to ensure flag that briefly disables this while merge+deploy es5 branchother use cases (api feature logs and phabricator) continue to work. * The completion suggester queries we issue to 2.x will not work on 5.x.## Phabricator should be able to point to codfw and do a rebuild from a maint script, then point the search at codfw, then do yet another rebuild to cover any updates that occured between the first rebuild and the switch over. ## Check with @Anomie about api feature log. Perhaps downtime is acceptable here? # Revert prior patch sending all traffic to codfw, This is a bigger problem and isallowing traffic to flow to eqiad again. There will be some sticking points that we may not easily solved.be able to get around, We may have to live with a brief service interruption while deploying the merged es5 branch and pushing out the config to switch read traffic to codfw.but should look into if there are good ways: * Need to double check that our production mappings all load into es5 correctly.* The completion suggester indices built by 2.x will sort-of work against 5.x, Mostly this is the string -> text field type rename,but not in an ideal manner. but there might be other issueWe may need some hackery in the code to detect which version of the titlesuggest index is being used and do different things with the results. ** titlesuggest may have drastically changed, need to test how this works.