Page MenuHomePhabricator
Create Task
Maniphest T234854

Upgrade ELK Stack
Open, MediumPublic
Actions

Description

Tracking task for upgrading the ELK stack to a more current stable release (targeting version 7.2)

High level items

  1. Build an ELK 7 upgrade environment in parallel to production
    • Provision ES 7 hosts (HW & OS)
    • Provision Logstash/Kibana 7 collector hosts (VM & OS)
    • Make new versions of ELK software installable via apt
    • Puppetize logging ES 7
    • Puppetize Logstash 7
    • Puppetize Kibana 7
    • Configure service address for load balanced Kibana frontend

2. Determine legal viability of amazon open distro for elasticsearch, if so
[] Integrate RBAC features with LDAP
[] Puppetize management of security users, roles, mappings, etc.

  1. Ingest production logs
    • Determine best way to handle/manage logstash plugins in the new version & execute
    • Consume from kafka-logging
    • Determine best method to bridge gap for ingesting log sources not not yet in Kafka
    • Validate log parsing, storage, etc.
    • Investigate and upgrade/adapt curator as necessary
    • Import Kibana configuration (saved searches, dashboards, visualizations, etc.)

4. Determine if alerting features should be enabled, if so...
[] document guidelines for alerting functionality

  1. Overall validation and cut over
    • Provide access to new environment widely, with old env still available as a backup. (https://logstash-next.wikimedia.org)
      • Gather/address bugs identified during this period (to be expanded as we gain better understanding/experience here)
    • Perform cut-over (name switch to logstash.wm.o)
  2. Migrate Kafka-logging brokers to ELK 7 cluster
  3. Fold (reimage/migrate) ELK 5 hardware into ELK7 cluster
  4. Retire ELK 5 VMs

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -2logstash: set v7 cluster to version 7.8
operations/puppetproduction+14 -1add thirdparty/elastic78 component
operations/puppetproduction+0 -12logstash: decom check_procs
operations/puppetproduction+11 -0elasticsearch: add max_clause_count setting
operations/puppetproduction+1 -1assign codfw logstash ssd hosts role::insetup
operations/puppetproduction+115 -7logstash: add ES 7 compatible logstash template
operations/puppetproduction+10 -0cache: map logstash-next.wikimedia.org and cas-logstash to kibana-next lvs
operations/dnsmaster+1 -0dns: add logstash-next.wikimedia.org record
operations/puppetproduction+24 -22add kibana-next SANs to kibana cert
operations/puppetproduction+1 -1lvs: kibana-next: promote from "service_setup" to "lvs_setup"
operations/puppetproduction+51 -0add load balancing for kibana-next
operations/puppetproduction+8 -0add profile::idp::client::httpd hiera for elk7 env
operations/puppetproduction+13 -0logstash::collector7 ingest deprecated logs from kafka
operations/puppetproduction+4 -2logstash: remove defalut value from kafka input type field
operations/puppetproduction+78 -0logstash: output logs ingested by deprecated inputs to kafka-logging
operations/dnsmaster+12 -0dns: add kibana-next and logstash-next service addresses
operations/puppetproduction+57 -0lvs: add entries for logstash-next and kibana-next
operations/puppetproduction+4 -2logstash: set kafka consumer groups at the role level
operations/puppetproduction+86 -74logstash: add kafka ssl_endpoint_identification_algorithm param
operations/puppetproduction+0 -79logstash: remove non-kafka inputs from elk7 cluster
operations/puppetproduction+517 -2logstash: create elk7 logstash collector profile
operations/puppetproduction+1 -0logstash: set es config_version to 7 on elk7 hosts
operations/puppetproduction+8 -3kibana: add kibana_package param and set elk7 hosts to -oss variant
operations/puppetproduction+3 -0logstash: add logstash_package param and set elk7 to -oss variant
operations/puppetproduction+147 -0logstash: create elk7 logstash role and assign to elk7 collectors
operations/puppetproduction+1 -1logstash: set elk7 es heap_memory to 24G
operations/puppetproduction+1 -0logstash: set elk7 es config_version to 7
operations/puppetproduction+27 -1elasticsearch: add buster openjdk 8 repository
operations/puppetproduction+85 -0logstash: create elk7 ES role and assign to elk7 ES hw hosts
operations/dnsmaster+12 -0add forwad/reverse entries for logstash 7 collector hosts
operations/puppetproduction+525 -19Introduce Elastic 7 support
operations/puppetproduction+5 -5aptrepo: include minor version in elastic 7 repos
operations/puppetproduction+6 -0install_server: use Buster for elastic 7 cluster
operations/puppetproduction+12 -0aptrepo: add elastic 7
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 25 2019, 7:17 PM

Change 552881 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: create elk7 logstash role and assign to elk7 collectors

https://gerrit.wikimedia.org/r/552881

gerritbot added a comment.Dec 2 2019, 3:13 PM

Change 552837 merged by Herron:
[operations/puppet@production] logstash: create elk7 ES role and assign to elk7 ES hw hosts

https://gerrit.wikimedia.org/r/552837

gerritbot added a comment.Dec 2 2019, 3:53 PM

Change 554095 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] elasticsearch: add buster openjdk 8 repository

https://gerrit.wikimedia.org/r/554095

gerritbot added a comment.Dec 2 2019, 4:11 PM

Change 554095 merged by Herron:
[operations/puppet@production] elasticsearch: add buster openjdk 8 repository

https://gerrit.wikimedia.org/r/554095

gerritbot added a comment.Dec 2 2019, 4:29 PM

Change 554101 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: set elk7 es config_version to 7

https://gerrit.wikimedia.org/r/554101

gerritbot added a comment.Dec 2 2019, 4:33 PM

Change 554101 merged by Herron:
[operations/puppet@production] logstash: set elk7 es config_version to 7

https://gerrit.wikimedia.org/r/554101

gerritbot added a comment.Dec 2 2019, 4:47 PM

Change 554103 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: set elk7 es heap_memory to 24G

https://gerrit.wikimedia.org/r/554103

gerritbot added a comment.Dec 2 2019, 4:48 PM

Change 554103 merged by Herron:
[operations/puppet@production] logstash: set elk7 es heap_memory to 24G

https://gerrit.wikimedia.org/r/554103

gerritbot added a comment.Dec 2 2019, 8:34 PM

Change 552881 merged by Herron:
[operations/puppet@production] logstash: create elk7 logstash role and assign to elk7 collectors

https://gerrit.wikimedia.org/r/552881

gerritbot added a comment.Dec 2 2019, 9:04 PM

Change 554152 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add logstash_package param and set elk7 to -oss variant

https://gerrit.wikimedia.org/r/554152

gerritbot added a comment.Dec 2 2019, 9:11 PM

Change 554152 merged by Herron:
[operations/puppet@production] logstash: add logstash_package param and set elk7 to -oss variant

https://gerrit.wikimedia.org/r/554152

gerritbot added a comment.Dec 2 2019, 9:21 PM

Change 554157 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kibana: add kibana_package param and set elk7 hosts to -oss variant

https://gerrit.wikimedia.org/r/554157

gerritbot added a comment.Dec 2 2019, 9:31 PM

Change 554157 merged by Herron:
[operations/puppet@production] kibana: add kibana_package param and set elk7 hosts to -oss variant

https://gerrit.wikimedia.org/r/554157

gerritbot added a comment.Dec 2 2019, 9:37 PM

Change 554160 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: set es config_version to 7 on elk7 hosts

https://gerrit.wikimedia.org/r/554160

gerritbot added a comment.Dec 2 2019, 9:39 PM

Change 554160 merged by Herron:
[operations/puppet@production] logstash: set es config_version to 7 on elk7 hosts

https://gerrit.wikimedia.org/r/554160

elukey added a subscriber: elukey.Dec 3 2019, 8:06 AM

Hello! I took the liberty to ack a lot of criticals/unknowns in icinga that were related to these new hosts, IIUC these are not in production :)

fgiunchedi added a comment.Dec 3 2019, 9:07 AM
In T234854#5708171, @elukey wrote:

Hello! I took the liberty to ack a lot of criticals/unknowns in icinga that were related to these new hosts, IIUC these are not in production :)

Thanks! I've downtimed the new hosts and their services until thurs

gerritbot added a comment.Dec 3 2019, 3:48 PM

Change 554314 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: create elk7 logstash collector profile

https://gerrit.wikimedia.org/r/554314

gerritbot added a comment.Dec 3 2019, 6:33 PM

Change 554314 merged by Herron:
[operations/puppet@production] logstash: create elk7 logstash collector profile

https://gerrit.wikimedia.org/r/554314

gerritbot added a comment.Dec 3 2019, 7:42 PM

Change 554355 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: remove non-kafka inputs from elk7 cluster

https://gerrit.wikimedia.org/r/554355

gerritbot added a comment.Dec 3 2019, 7:46 PM

Change 554355 merged by Herron:
[operations/puppet@production] logstash: remove non-kafka inputs from elk7 cluster

https://gerrit.wikimedia.org/r/554355

gerritbot added a comment.Dec 3 2019, 8:32 PM

Change 554362 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add kafka ssl_endpoint_identification_algorithm param

https://gerrit.wikimedia.org/r/554362

gerritbot added a comment.Dec 3 2019, 8:50 PM

Change 554362 merged by Herron:
[operations/puppet@production] logstash: add kafka ssl_endpoint_identification_algorithm param

https://gerrit.wikimedia.org/r/554362

gerritbot added a comment.Dec 4 2019, 9:14 AM

Change 554472 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: set kafka consumer groups at the role level

https://gerrit.wikimedia.org/r/554472

gerritbot added a comment.Dec 4 2019, 9:22 AM

Change 554472 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: set kafka consumer groups at the role level

https://gerrit.wikimedia.org/r/554472

herron updated the task description. (Show Details)Dec 4 2019, 3:35 PM
Stashbot added a comment.Dec 5 2019, 8:03 AM

Mentioned in SAL (#wikimedia-operations) [2019-12-05T08:03:52Z] <elukey> remove logstash_cleanup_indices_apifeatureusage-search.svc.codfw.wmnet and logstash_cleanup_indices_apifeatureusage-search.svc.eqiad.wmnet from logstash1025,logstash1024,logstash1023,logstash2024,logstash2025 to reduce cronspam - T234854

elukey added a comment.Dec 5 2019, 8:08 AM

Hello :)

From cronspam I can see two errors that happen daily:

Cron[logstash_cleanup_indices_logstash]

elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: Elasticsearch version 7.4.2 incompatible with this version of Curator (5.2.0)
Cron[logstash_cleanup_indices_apifeatureusage-search.svc.codfw.wmnet]
Cron[logstash_cleanup_indices_apifeatureusage-search.svc.eqiad.wmnet]

Error: Invalid value for "--config": Path "/etc/curator/config-apifeatureusage-search.svc.codfw.wmnet.yaml" does not exist.

The latter seems fixed, I removed the crontab entries (since I didn't find anything in puppet related to the hosts sending cronspam) and ran puppet to make sure that those crons were not on the catalog. The former seems to be something to check/review :)

gerritbot added a comment.Dec 5 2019, 4:12 PM
Change 554905 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] lvs: add entries for logstash-next and kibana-next



https://gerrit.wikimedia.org/r/554905
gerritbot added a project: Patch-For-Review.Dec 5 2019, 4:12 PM
Change 554906 had a related patch set uploaded (by Herron; owner: Herron):

[operations/dns@master] dns: add kibana-next and logstash-next service addresses



https://gerrit.wikimedia.org/r/554906
gerritbot added a comment.Dec 9 2019, 6:34 PM
Change 554905 merged by Herron:

[operations/puppet@production] lvs: add entries for logstash-next and kibana-next



https://gerrit.wikimedia.org/r/554905
gerritbot added a comment.Dec 9 2019, 6:50 PM
Change 554906 merged by Herron:

[operations/dns@master] dns: add kibana-next and logstash-next service addresses



https://gerrit.wikimedia.org/r/554906
elukey added a comment.Dec 10 2019, 8:08 AM
@herron hello :) Any comment on what I wrote above about cronspam?
herron added a comment.Dec 10 2019, 12:27 PM
@elukey hey, yes that's been fixed by making a newer version of curator available to the new clusters.  Haven't seen cron errors from these since Dec 5.  Thanks for cleaning up the "config does not exist" entries!
gerritbot added a comment.Feb 11 2020, 6:18 PM
Change 571548 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] logstash: output logs ingested by deprecated inputs to kafka-logging



https://gerrit.wikimedia.org/r/571548
gerritbot added a comment.Feb 11 2020, 6:33 PM
Change 571554 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] logstash::collector7 ingest deprecated logs from kafka



https://gerrit.wikimedia.org/r/571554
gerritbot added a comment.Feb 12 2020, 2:11 AM
Change 571622 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] logstash: add ES 7 compatible logstash template



https://gerrit.wikimedia.org/r/571622
gerritbot added a comment.Feb 12 2020, 9:37 PM
Change 571813 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] logstash: remove defalut value from kafka input type field



https://gerrit.wikimedia.org/r/571813
gerritbot added a comment.Feb 18 2020, 4:22 PM
Change 571548 merged by Herron:

[operations/puppet@production] logstash: output logs ingested by deprecated inputs to kafka-logging



https://gerrit.wikimedia.org/r/571548
gerritbot added a comment.Feb 18 2020, 7:22 PM
Change 571813 merged by Herron:

[operations/puppet@production] logstash: remove defalut value from kafka input type field



https://gerrit.wikimedia.org/r/571813
gerritbot added a comment.Feb 18 2020, 8:07 PM
Change 571554 merged by Herron:

[operations/puppet@production] logstash::collector7 ingest deprecated logs from kafka



https://gerrit.wikimedia.org/r/571554
gerritbot added a comment.Feb 25 2020, 9:04 PM
Change 574862 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] add load balancing for kibana-next



https://gerrit.wikimedia.org/r/574862
gerritbot added a comment.Feb 27 2020, 6:12 PM
Change 575320 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] add profile::idp::client::httpd hiera for elk7 env



https://gerrit.wikimedia.org/r/575320
gerritbot added a comment.Feb 27 2020, 6:48 PM
Change 575320 merged by Herron:

[operations/puppet@production] add profile::idp::client::httpd hiera for elk7 env



https://gerrit.wikimedia.org/r/575320
gerritbot added a comment.Feb 28 2020, 3:24 PM
Change 574862 merged by Herron:

[operations/puppet@production] add load balancing for kibana-next



https://gerrit.wikimedia.org/r/574862
gerritbot added a comment.Feb 28 2020, 10:34 PM
Change 575631 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] lvs: kibana-next: promote from "service_setup" to "lvs_setup"



https://gerrit.wikimedia.org/r/575631
gerritbot added a comment.Mar 2 2020, 3:14 PM
Change 575631 merged by Herron:

[operations/puppet@production] lvs: kibana-next: promote from "service_setup" to "lvs_setup"



https://gerrit.wikimedia.org/r/575631
gerritbot added a comment.Mar 2 2020, 9:45 PM
Change 576152 had a related patch set uploaded (by Herron; owner: Herron):

[operations/dns@master] dns: add logstash-next.wikimedia.org record



https://gerrit.wikimedia.org/r/576152
gerritbot added a comment.Mar 2 2020, 9:45 PM
Change 576151 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] cache: map logstash-next.wikimedia.org to kibana-next lvs



https://gerrit.wikimedia.org/r/576151
gerritbot added a comment.Mar 3 2020, 6:40 PM
Change 576411 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] add kibana-next SANs to kibana cert



https://gerrit.wikimedia.org/r/576411
gerritbot added a comment.Mar 3 2020, 6:49 PM
Change 576411 merged by Herron:

[operations/puppet@production] add kibana-next SANs to kibana cert



https://gerrit.wikimedia.org/r/576411
gerritbot added a comment.Mar 3 2020, 10:37 PM
Change 576152 abandoned by Herron:

dns: add logstash-next.wikimedia.org record



Reason:

abandoning in favor of a5257d4fc7826c26a6a7e60799b1c71fc789ed65



https://gerrit.wikimedia.org/r/576152
gerritbot added a comment.Mar 4 2020, 3:57 PM
Change 576151 merged by Herron:

[operations/puppet@production] cache: map logstash-next.wikimedia.org and cas-logstash to kibana-next lvs



https://gerrit.wikimedia.org/r/576151
gerritbot added a comment.Mar 4 2020, 9:39 PM
Change 576967 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] elasticsearch: add max_clause_count setting



https://gerrit.wikimedia.org/r/576967
herron updated the task description. (Show Details)Mar 5 2020, 5:37 PM
gerritbot added a comment.Mar 11 2020, 7:35 PM
Change 571622 merged by Herron:

[operations/puppet@production] logstash: add ES 7 compatible logstash template



https://gerrit.wikimedia.org/r/571622
gerritbot added a comment.Mar 13 2020, 3:45 AM
Change 579461 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] assign codfw logstash ssd hosts role::insetup



https://gerrit.wikimedia.org/r/579461
gerritbot added a comment.Mar 13 2020, 3:49 AM
Change 579461 merged by Herron:

[operations/puppet@production] assign codfw logstash ssd hosts role::insetup



https://gerrit.wikimedia.org/r/579461
gerritbot added a comment.Mar 17 2020, 7:01 PM
Change 576967 abandoned by Herron:

elasticsearch: add max_clause_count setting



Reason:

going with I2e690d26e5bd4d9961f261eb049f33ef58ad2588 instead



https://gerrit.wikimedia.org/r/576967
herron updated the task description. (Show Details)Mar 30 2020, 3:39 PM
Krinkle added a subscriber: Krinkle.EditedMar 31 2020, 6:29 PM
First impressions of the new Logstash/Kibana based on using Firefox 74 for macOS on an idle high-end MacBook Pro using a fast WiFi connection.



    

  • It is even slower to load. Just to have the UI appear initially at all now takes 7-8 seconds on logstash-next compared to ~ 1s second on logstash (this is while loading the domain and seeing the "Loading" animation).
    • 




    

  • All interface links and buttons are unresposive. When hovering any link or button (e.g. on any dashboard the "Close", "Show dates", "Lucene" or "Refresh" buttons) they are without a pointer cursor for the first 1-2 seconds before they can be clicked.
      

    • This also applies to modal interfaces such as the "edit filter" overlay, and the date inputs.
      • 

    • This is actually really difficult to screw up in a modern browser, so I'm kind of impressed they managed to make the UI this bad.
      • 

    • 




    

  • As a silver lining, they seem to have finally fixed the autocomplete widget for "Edit filter". It no longer tries to preload all 90 days of Logstash indexes client-side and iterate over every unique field on every keystroke (which is what led to T189333). Instead, this data is now lazy-loaded in chunks and filtering is debounced properly, resulting in an input field that is now actually usable, in all browsers I tried. Yay!
    • 

elukey added a comment.Apr 24 2020, 8:10 AM
The codfw cluster is currently yellow, from explain I see a lot of "explanation" : "node does not match index setting [index.routing.allocation.require] filters [disktype:\"hdd\"]"



I acked the alerts since the notifications were turned off.
colewhite added a comment.Apr 24 2020, 7:10 PM
there was some work to rotate old indexes to spinning disks but the cluster knew of no nodes with the "hdd" disktype attribute.  it looks like the configuration was stale and restarting logstash[2021-2022] allowed the indexes to be assigned.
brennen added a subscriber: brennen.May 4 2020, 5:31 PM
fgiunchedi added a comment.May 22 2020, 9:26 AM
Since Elastic stack 7.7 has been released I think it'd make sense we upgrade to that before the switch, supposedly there have been improvements to memory usage!
fgiunchedi added a subtask: Restricted Task.Jun 22 2020, 9:36 AM
gerritbot added a comment.Jul 3 2020, 9:11 AM
Change 609397 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):

[operations/puppet@production] logstash: decom check_procs



https://gerrit.wikimedia.org/r/609397
gerritbot added a comment.Jul 7 2020, 1:37 PM
Change 609397 merged by Filippo Giunchedi:

[operations/puppet@production] logstash: decom check_procs



https://gerrit.wikimedia.org/r/609397
gerritbot added a comment.Jul 7 2020, 3:12 PM
Change 610079 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] add thirdparty/elastic78 component



https://gerrit.wikimedia.org/r/610079
gerritbot added a comment.Jul 7 2020, 3:51 PM
Change 610079 merged by Herron:

[operations/puppet@production] add thirdparty/elastic78 component



https://gerrit.wikimedia.org/r/610079
gerritbot added a comment.Jul 7 2020, 6:45 PM
Change 610135 had a related patch set uploaded (by Herron; owner: Herron):

[operations/puppet@production] logstash: set v7 cluster to version 7.8



https://gerrit.wikimedia.org/r/610135
gerritbot added a comment.Jul 9 2020, 4:26 PM
Change 610135 merged by Herron:

[operations/puppet@production] logstash: set v7 cluster to version 7.8



https://gerrit.wikimedia.org/r/610135
Stashbot added a comment.Jul 9 2020, 7:16 PM
Mentioned in SAL (#wikimedia-operations) [2020-07-09T19:16:27Z] <herron> upgraded eqiad elk7 cluster from 7.4.2 to 7.8.0 T234854
jcrespo added a subscriber: jcrespo.Wed, Jul 29, 12:46 PM
I am getting a lot of 500 internal server errors on logstash-next instance. I am guessing that is expected/WIP?
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL