Page MenuHomePhabricator
Create Task
Maniphest T308676

Elasticsearch 7.10.2 rollout plan
Closed, ResolvedPublic
Actions

Description

Prerequisites:

Schedule:

  • Starting week: Aug 29, 2022
  • Train version: Expecting 1.39.0-wmf.28 with branch cut on Sept 5.

Once all prerequisites are verified define the week starting date for the rollout:

Plan:

  • Week 1
    • Monday: Upgrade cloudelastic to elasticsearch 7.10.2 and verify that updates are flowing properly
    • Tuesday: Merge apifeatureusage prep. Start the rollout on codfw and monitor updates
    • After the branch cut, merge:
      • Merge the es710 branch into master in vendor, CirrusSearch, and Elastica repositories
    • Note the mediawiki version they'll land into and update the mw-config patch (https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/824787) with it
    • Add these patches as risky in a comment to the corresponding deployment blockers phab task
    • Soon after merging upgrade deployment-prep to elasticsearch 7.10.2 and verify search functions/error logs
    • Once codfw is fully upgraded deploy the mw-config patch to switch search traffic based on the mw train version
  • Week 2
    • Wait for the train to rollout everywhere and verify search functions/error logs, search traffic will start flowing to codfw as the train rolls forward, be available to releng when syncing wiki groups
  • Week 3 (after the expected train version is running everywhere and unlikely to be rolled back)

Things to watch out:

  • CirrusSearch maintenance scripts: completion suggester, dump index, saneitizer

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcausse updated the task description. (Show Details)May 19 2022, 8:16 AM
dcausse updated the task description. (Show Details)May 19 2022, 2:21 PM
dcausse updated the task description. (Show Details)May 19 2022, 6:17 PM
dcausse updated the task description. (Show Details)May 19 2022, 6:28 PM
MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.May 23 2022, 3:29 PM
MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.
MPhamWMF moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.May 23 2022, 3:39 PM
bd808 subscribed.Jun 1 2022, 9:45 PM
bd808 mentioned this in T311787: Upgrade elasticsearch python client libraries after production service.Jun 30 2022, 9:32 PM
EBernhardson updated the task description. (Show Details)Jul 25 2022, 4:47 PM
EBernhardson updated the task description. (Show Details)
EBernhardson mentioned this in T313434: Migrate apifeatureusage to elasticsearch 7.Jul 26 2022, 5:22 PM
MPhamWMF assigned this task to EBernhardson.Aug 15 2022, 3:11 PM
EBernhardson updated the task description. (Show Details)Aug 16 2022, 6:44 PM
EBernhardson updated the task description. (Show Details)
EBernhardson updated the task description. (Show Details)Aug 17 2022, 8:59 PM
EBernhardson updated the task description. (Show Details)
bking updated the task description. (Show Details)Aug 17 2022, 9:29 PM
EBernhardson updated the task description. (Show Details)Aug 19 2022, 5:55 PM
EBernhardson updated the task description. (Show Details)Aug 22 2022, 5:17 PM
EBernhardson updated the task description. (Show Details)Aug 22 2022, 5:21 PM
gerritbot added a comment.Aug 23 2022, 7:08 PM

Change 825874 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] elastic: clear old es_6 resources during upgrade

https://gerrit.wikimedia.org/r/825874

gerritbot added a project: Patch-For-Review.Aug 23 2022, 7:09 PM
gerritbot added a comment.Aug 23 2022, 8:25 PM

Change 825883 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: es7 removed bulk threadpool

https://gerrit.wikimedia.org/r/825883

gerritbot added a comment.Aug 24 2022, 7:12 PM

Change 825874 merged by Bking:

[operations/cookbooks@master] elastic: clear old es_6 resources during upgrade

https://gerrit.wikimedia.org/r/825874

gerritbot added a comment.Aug 24 2022, 8:16 PM

Change 825883 merged by Bking:

[operations/puppet@production] elastic: es7 removed bulk threadpool

https://gerrit.wikimedia.org/r/825883

Maintenance_bot removed a project: Patch-For-Review.Aug 24 2022, 8:30 PM
gerritbot added a comment.Aug 24 2022, 8:43 PM

Change 826383 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] opensearch: replace outdated config

https://gerrit.wikimedia.org/r/826383

gerritbot added a project: Patch-For-Review.Aug 24 2022, 8:43 PM
gerritbot added a comment.Aug 24 2022, 9:06 PM

Change 826386 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: use our jvm not elasticsearch's jvm

https://gerrit.wikimedia.org/r/826386

gerritbot added a comment.Aug 24 2022, 9:33 PM

Change 826386 merged by Bking:

[operations/puppet@production] elastic: use our jvm not elasticsearch's jvm

https://gerrit.wikimedia.org/r/826386

gerritbot added a comment.Aug 24 2022, 10:37 PM

Change 826396 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: don't start es7 unit until we tell it

https://gerrit.wikimedia.org/r/826396

gerritbot added a comment.Aug 25 2022, 5:36 PM

Change 826396 merged by Bking:

[operations/puppet@production] elastic: don't start es7 unit until we tell it

https://gerrit.wikimedia.org/r/826396

gerritbot added a comment.Aug 25 2022, 6:54 PM

Change 826640 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] elastic: fix string concatenation

https://gerrit.wikimedia.org/r/826640

gerritbot added a comment.Aug 25 2022, 7:29 PM

Change 826647 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] elastic: no need to run puppet during es 7 upgrade

https://gerrit.wikimedia.org/r/826647

gerritbot added a comment.Aug 25 2022, 7:37 PM

Change 826647 merged by jenkins-bot:

[operations/cookbooks@master] elastic: no need to run puppet during es 7 upgrade

https://gerrit.wikimedia.org/r/826647

gerritbot added a comment.Aug 25 2022, 8:00 PM

Change 826640 merged by jenkins-bot:

[operations/cookbooks@master] elastic: fix string concatenation

https://gerrit.wikimedia.org/r/826640

gerritbot added a comment.Aug 25 2022, 8:21 PM

Change 826651 had a related patch set uploaded (by Bking; author: Bking):

[operations/cookbooks@master] elastic: use correct systemd command

https://gerrit.wikimedia.org/r/826651

gerritbot added a comment.Aug 25 2022, 8:26 PM

Change 826651 merged by jenkins-bot:

[operations/cookbooks@master] elastic: use correct systemd command

https://gerrit.wikimedia.org/r/826651

gerritbot added a comment.Aug 29 2022, 6:52 PM

Change 827204 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/Elastica@master] Revert "Revert "Switch to Elastica 7.1.5""

https://gerrit.wikimedia.org/r/827204

EBernhardson updated the task description. (Show Details)Aug 29 2022, 6:53 PM
gerritbot added a comment.Aug 29 2022, 6:57 PM

Change 827544 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/vendor@master] Upgrade to Elastica 7.1.5

https://gerrit.wikimedia.org/r/827544

gerritbot added a comment.Aug 29 2022, 6:57 PM

Change 827545 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Merge remote-tracking branch 'gerrit/es710'

https://gerrit.wikimedia.org/r/827545

EBernhardson updated the task description. (Show Details)Aug 29 2022, 6:58 PM
EBernhardson updated the task description. (Show Details)
bking subscribed.Aug 29 2022, 9:16 PM
gerritbot added a comment.Aug 30 2022, 8:33 PM

Change 828086 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@es710] Merge remote-tracking branch 'gerrit/es710' into es710

https://gerrit.wikimedia.org/r/828086

gerritbot added a comment.Aug 30 2022, 8:34 PM

Change 828086 abandoned by Ebernhardson:

[mediawiki/extensions/CirrusSearch@es710] Merge remote-tracking branch 'gerrit/es710' into es710

Reason:

should be uploaded to master branch, not es710

https://gerrit.wikimedia.org/r/828086

gerritbot added a comment.Aug 30 2022, 8:37 PM

Change 827545 merged by Ebernhardson:

[mediawiki/extensions/CirrusSearch@master] Merge remote-tracking branch 'gerrit/es710' into es710

https://gerrit.wikimedia.org/r/827545

gerritbot added a comment.Aug 30 2022, 8:37 PM

Change 827544 merged by Ebernhardson:

[mediawiki/vendor@master] Upgrade to Elastica 7.1.5

https://gerrit.wikimedia.org/r/827544

gerritbot added a comment.Aug 30 2022, 8:37 PM

Change 827204 merged by Ebernhardson:

[mediawiki/extensions/Elastica@master] Switch to Elastica 7.1.5 [re-apply]

https://gerrit.wikimedia.org/r/827204

Jdforrester-WMF updated the task description. (Show Details)Aug 30 2022, 8:42 PM
ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.28; 2022-09-05).Aug 30 2022, 9:00 PM
EBernhardson updated the task description. (Show Details)Aug 30 2022, 10:10 PM
gerritbot added a comment.Aug 31 2022, 7:06 PM

Change 828403 had a related patch set uploaded (by Ryan Kemper; author: DCausse):

[operations/puppet@production] Relax elasticsearch master node detection

https://gerrit.wikimedia.org/r/828403

gerritbot added a comment.Aug 31 2022, 7:06 PM

Change 828403 merged by Ryan Kemper:

[operations/puppet@production] Relax elasticsearch master node detection

https://gerrit.wikimedia.org/r/828403

dcausse updated the task description. (Show Details)Sep 6 2022, 6:08 AM
EBernhardson added a comment.Sep 6 2022, 4:52 PM

Rollout is currently progressing. Found one regression from logs:

Search backend error during comp_suggest search for 'mediaw' after 131: x_content_parse_exception: [1:516] [terms_lookup] unknown field [1]

Looks to be an issue with a php array being encoded to a json object. Patch (https://gerrit.wikimedia.org/r/c/830214/) scheduled for backport window today. Otherwise it all looks reasonable so far.

EBernhardson added a subtask: T317381: Reduction in helpfulness and quantity of autocomplete search results.Sep 9 2022, 5:29 AM
EBernhardson added a subtask: T317374: Search now caps total results count at 10k because of elasticsearch 7 upgrade.
dcausse updated the task description. (Show Details)Sep 9 2022, 1:30 PM
RKemper changed the status of subtask T317374: Search now caps total results count at 10k because of elasticsearch 7 upgrade from Open to In Progress.Sep 9 2022, 7:12 PM
Pols12 mentioned this in T101236: TTMServer performance and coverage issues.Sep 10 2022, 3:45 PM
kostajh mentioned this in T317187: GrowthExperiments Special:Homepage: investigate performance regression since September 6 2022.Sep 12 2022, 11:35 AM
kostajh subscribed.
RKemper changed the status of subtask T317686: Upgrade eqiad cluster to Elasticsearch 7.10.2 from Open to In Progress.Sep 13 2022, 5:43 PM
EBernhardson updated the task description. (Show Details)Sep 14 2022, 7:23 PM

eqiad is now running elastic 7 as well. The completion indices will rebuild overnight, we should be able to verify them tomorrow and switch traffic back to it's local datacenter instead of all being routed to codfw.

bking closed subtask T317686: Upgrade eqiad cluster to Elasticsearch 7.10.2 as Resolved.Sep 14 2022, 8:01 PM
gerritbot added a comment.Sep 15 2022, 2:26 PM

Change 832323 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/mediawiki-config@master] Revert "cirrus: Handle transition to elasticsearch 7.10"

https://gerrit.wikimedia.org/r/832323

dcausse updated the task description. (Show Details)Sep 15 2022, 2:51 PM
gerritbot added a comment.Sep 15 2022, 8:07 PM

Change 832323 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "cirrus: Handle transition to elasticsearch 7.10"

https://gerrit.wikimedia.org/r/832323

Stashbot added a comment.Sep 15 2022, 8:07 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-15T20:07:40Z] <thcipriani@deploy1002> Started scap: Backport for [[gerrit:832323|Revert "cirrus: Handle transition to elasticsearch 7.10" (T308676)]]

Stashbot added a comment.Sep 15 2022, 8:08 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-15T20:08:00Z] <thcipriani@deploy1002> thcipriani and dcausse: Backport for [[gerrit:832323|Revert "cirrus: Handle transition to elasticsearch 7.10" (T308676)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet

Stashbot added a comment.Sep 15 2022, 8:15 PM

Mentioned in SAL (#wikimedia-operations) [2022-09-15T20:15:19Z] <thcipriani@deploy1002> Finished scap: Backport for [[gerrit:832323|Revert "cirrus: Handle transition to elasticsearch 7.10" (T308676)]] (duration: 07m 39s)

gerritbot added a comment.Sep 15 2022, 8:21 PM

Change 826383 merged by Ryan Kemper:

[operations/puppet@production] opensearch: replace outdated config

https://gerrit.wikimedia.org/r/826383

Gehel closed subtask T317374: Search now caps total results count at 10k because of elasticsearch 7 upgrade as Resolved.Sep 16 2022, 2:59 PM
Izno subscribed.Sep 16 2022, 5:08 PM
Gehel closed subtask T317381: Reduction in helpfulness and quantity of autocomplete search results as Resolved.Sep 16 2022, 8:45 PM
Gehel closed subtask T316719: Upgrade codfw cluster to Elasticsearch 7.10.2 as Resolved.Sep 17 2022, 8:46 PM
Gehel closed subtask T316240: Upgrade deployment-prep to Elasticsearch 7.10.2 as Resolved.
EBernhardson updated the task description. (Show Details)Sep 19 2022, 3:56 PM
EBernhardson moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Sep 19 2022, 4:52 PM
Gehel closed subtask T316159: Upgrade cloudelastic cluster to 7.10.2 as Resolved.Sep 26 2022, 9:32 AM
Gehel closed subtask T315604: Upgrade relforge cluster to 7.10.2 as Resolved.Sep 26 2022, 9:50 AM
Gehel closed this task as Resolved.Sep 26 2022, 10:38 AM
Gehel subscribed.

Update has been sent to discovery@ and wikitech-l@. This can be closed.

gerritbot added a comment.Oct 20 2022, 9:06 PM

Change 845054 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: don't block on /root/allow_es7 existing

https://gerrit.wikimedia.org/r/845054

gerritbot added a comment.Oct 20 2022, 9:08 PM

Change 845055 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/cookbooks@master] elastic: no more /root/allow_es7

https://gerrit.wikimedia.org/r/845055

gerritbot added a comment.Oct 20 2022, 9:09 PM

Change 845054 merged by Ryan Kemper:

[operations/puppet@production] elastic: don't block on /root/allow_es7 existing

https://gerrit.wikimedia.org/r/845054

gerritbot added a comment.Oct 20 2022, 11:09 PM

Change 845055 merged by Ryan Kemper:

[operations/cookbooks@master] elastic: no more /root/allow_es7

https://gerrit.wikimedia.org/r/845055

RKemper mentioned this in rCCKBbe11ce9e4555: elastic: clear old es_6 resources during upgrade.Dec 14 2022, 3:31 PM
RKemper mentioned this in rCCKB482419e70f45: elastic: no need to run puppet during es 7 upgrade.
RKemper mentioned this in rCCKBdbd6c610ccc2: elastic: fix string concatenation.
bking mentioned this in rCCKB77d2482175f7: elastic: use correct systemd command.Dec 14 2022, 3:31 PM
bking mentioned this in rCCKB3e70ebb746d6: elastic: no more /root/allow_es7.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL