Page MenuHomePhabricator
Create Task
Maniphest T167636

Investigate perf regression after elasticsearch 5.3.2 deployment
Closed, ResolvedPublic
Actions

Description

Elasticsearch 5.3.2 seems to have caused a visible perf regression on query percentiles:

  • fulltext: +20ms
  • morelike: +40ms
  • compsuggest: +2ms

Grafana - Elasticsearch Percentiles - Google Chrome_009.png (1×2 px, 310 KB)

Young GC activity seems to have jumped as well while the amount of heap used seems to have decreased:

Grafana - Elasticsearch Memory - Google Chrome_010.png (535×2 px, 204 KB)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
elasticsearch: remove UseConcMarkSweepGCoperations/puppetproduction+4 -3
Customize query in gerrit

Event Timeline

dcausse created this task.Jun 12 2017, 8:34 AM
Restricted Application added projects: Discovery-ARCHIVED, Discovery-Search. · View Herald TranscriptJun 12 2017, 8:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse added subscribers: EBernhardson, Gehel.Jun 12 2017, 8:35 AM
Gehel closed this task as a duplicate of T167685: performance regression after upgrading elasticsearch to v5.3.2.Jun 12 2017, 2:53 PM
Gehel reopened this task as Open.Jun 12 2017, 2:56 PM
gerritbot subscribed.Jun 12 2017, 2:56 PM

Change 358383 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: remove UseConcMarkSweepGC

https://gerrit.wikimedia.org/r/358383

gerritbot added a project: Patch-For-Review.Jun 12 2017, 2:56 PM
Gehel triaged this task as High priority.Jun 12 2017, 2:57 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.Jun 12 2017, 3:38 PM
Gehel moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.
Stashbot subscribed.Jun 13 2017, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2017-06-13T14:22:28Z] <gehel> restarting elasticsearch on relforge to validate GC configuration - T167636

Stashbot added a comment.Jun 13 2017, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2017-06-13T15:09:12Z] <gehel> applying new GC configuration on elastic1018 - T167636

Gehel added a comment.Jun 14 2017, 4:15 PM

elastic1018 is looking good, with significantly lower GC times than other nodes (see grafana). Next test is to roll out to the whole cluster...

gerritbot added a comment.Jun 15 2017, 8:29 AM

Change 358383 merged by Gehel:
[operations/puppet@production] elasticsearch: remove UseConcMarkSweepGC

https://gerrit.wikimedia.org/r/358383

Gehel moved this task from not in use - please delete to Needs Reporting on the Discovery-Search (Current work) board.Jun 15 2017, 8:30 AM

Change is merged. It will require a full cluster restart to be taken into account before we can actually close it. Cluster restart is planned to start on Monday June 19th.

• debt closed this task as Resolved.Jun 16 2017, 5:32 PM
• debt claimed this task.
• debt subscribed.

Resolving, final restart of the clusters will happen on Monday, June 19, 2017

• debt removed • debt as the assignee of this task.Jun 19 2017, 6:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits