Change Details

Similar to the 1.x -> 2.x upgrade, we need to come up with a plan for how we will do the production migration. Likely this can be very similar to the last major upgrade: # Verify (or add necessary flags) the current codebase and the es5 branch can write to a cluster running the other version of elasticseach # Upgrade codfw to 5.x (hopefully one day activity, one a monday) ## Ship patch to stop sending writes to codfw. Note the time (SAL log or whatever) ## Shut down all hosts in codfw. ## Ship puppet patch to update elasticsearch.yml for 5.x ## Install new elasticsearch.deb ## Bring up the master capable nodes, let them elect a master ## Bring up the rest of the nodes, wait for green ## Reindex the timespan codfw was down. May need some hackishness to let the 2.x maint script talk to 5.x (just the version check) ## Somehow we need to build the 5.x completion suggester indices before the # Update mediawiki-config to send all search queries for next train deployment to codfw cluster. See https://gerrit.wikimedia.org/r/291257 for how we did this last time. Don't forget about TTMServer # Merge es5 branch into master and let it ride the train # After train has rolled forward to all wikis perform the same upgrade actions we did to codfw to eqiad. This will require more coordination to ensure other use cases (api feature logs and phabricator) continue to work. ## Phabricator should be able to point to codfw and do a rebuild from a maint script, then point the search at codfw, then do yet another rebuild to cover any updates that occured between the first rebuild and the switch over. ## Check with @Anomie about api feature log. Perhaps downtime is acceptable here? # Revert prior patch sending all traffic to codfw, allowing traffic to flow to eqiad again. There will be some sticking points that we may not easily be able to get around, but should look into if there are good ways: * The completion suggester indices built by 2.x will sort-of work against 5.x, but not in an ideal manner. We may need some hackery in the code to detect which version of the titlesuggest index is being used and do different things with the results.

Similar to the 1.x -> 2.x upgrade, we need to come up with a plan for how we will do the production migration. Likely this can be very similar to the last major upgrade: # Verify (or add necessary flags) the current codebase and the es5 branch can write to a cluster running the other version of elasticseach # Upgrade codfw to 5.x (hopefully one day activity, one a monday) ## Ship patch to stop sending writes to codfw. Note the time (SAL log or whatever) ## Shut down all hosts in codfw. ## Ship puppet patch to update elasticsearch.yml for 5.x ## Install new elasticsearch.deb ## Bring up the master capable nodes, let them elect a master ## Bring up the rest of the nodes, wait for green ## Reindex the timespan codfw was down. May need some hackishness to let the 2.x maint script talk to 5.x (just the version check) ## Somehow we need to build the 5.x completion suggester indices before the # Update mediawiki-config to send all search queries for next train deployment to codfw cluster. See https://gerrit.wikimedia.org/r/291257 for how we did this last time. Don't forget about TTMServer ## This patch should probably also disable the completion suggester. Serving autocomplete with titlesuggest indices built under 2.x will return undesirable results. # Merge es5 branch into master and let it ride the train # As the train rolls forward rebuild titlesuggest indices with 5.x. After all indices have been rebuilt re-enable completion search. # After train has rolled forward to all wikis perform the same upgrade actions we did to codfw to eqiad. This will require more coordination to ensure other use cases (api feature logs and phabricator) continue to work. ## Phabricator should be able to point to codfw and do a rebuild from a maint script, then point the search at codfw, then do yet another rebuild to cover any updates that occured between the first rebuild and the switch over. ## Check with @Anomie about api feature log. Perhaps downtime is acceptable here? # Revert prior patch sending all traffic to codfw, allowing traffic to flow to eqiad again. There will be some sticking points that we may not easily be able to get around, but should look into if there are good ways: * The completion suggester indices built by 2.x will sort-of work against 5.x, but not in an ideal manner. We may need some hackery in the code to detect which version of the titlesuggest index is being used and do different things with the results.

Similar to the 1.x -> 2.x upgrade, we need to come up with a plan for how we will do the production migration. Likely this can be very similar to the last major upgrade: # Verify (or add necessary flags) the current codebase and the es5 branch can write to a cluster running the other version of elasticseach # Upgrade codfw to 5.x (hopefully one day activity, one a monday) ## Ship patch to stop sending writes to codfw. Note the time (SAL log or whatever) ## Shut down all hosts in codfw. ## Ship puppet patch to update elasticsearch.yml for 5.x ## Install new elasticsearch.deb ## Bring up the master capable nodes, let them elect a master ## Bring up the rest of the nodes, wait for green ## Reindex the timespan codfw was down. May need some hackishness to let the 2.x maint script talk to 5.x (just the version check) ## Somehow we need to build the 5.x completion suggester indices before the # Update mediawiki-config to send all search queries for next train deployment to codfw cluster. See https://gerrit.wikimedia.org/r/291257 for how we did this last time. Don't forget about TTMServer ## This patch should probably also disable the completion suggester. Serving autocomplete with titlesuggest indices built under 2.x will return undesirable results. # Merge es5 branch into master and let it ride the train # As the train rolls forward rebuild titlesuggest indices with 5.x. After all indices have been rebuilt re-enable completion search. # After train has rolled forward to all wikis perform the same upgrade actions we did to codfw to eqiad. This will require more coordination to ensure other use cases (api feature logs and phabricator) continue to work. ## Phabricator should be able to point to codfw and do a rebuild from a maint script, then point the search at codfw, then do yet another rebuild to cover any updates that occured between the first rebuild and the switch over. ## Check with @Anomie about api feature log. Perhaps downtime is acceptable here? # Revert prior patch sending all traffic to codfw, allowing traffic to flow to eqiad again. There will be some sticking points that we may not easily be able to get around, but should look into if there are good ways: * The completion suggester indices built by 2.x will sort-of work against 5.x, but not in an ideal manner. We may need some hackery in the code to detect which version of the titlesuggest index is being used and do different things with the results.