Tracking task for upgrading the ELK stack to a more current stable release (targeting version 7.2)
High level items
1. Build an ELK 7 upgrade environment in parallel to production
[x] Provision ES 7 hosts (HW & OS)
[x] Provision Logstash/Kibana 7 collector hosts (VM & OS)
[x] Make new versions of ELK software installable via apt
[x] Puppetize logging ES 7
[x] Puppetize Logstash 7
[x] Puppetize Kibana 7
[x] Configure service address for load balanced Kibana frontend
~~2. Determine legal viability of amazon open distro for elasticsearch, if so~~
~~[] Integrate RBAC features with LDAP~~
~~[] Puppetize management of security users, roles, mappings, etc.~~
3. Ingest production logs
[x] Determine best way to handle/manage logstash plugins in the new version & execute
[x] Consume from kafka-logging
[x] Determine best method to bridge gap for ingesting log sources not not yet in Kafka
[x] Validate log parsing, storage, etc.
[x] Investigate and upgrade/adapt curator as necessary
[x] Import Kibana configuration (saved searches, dashboards, visualizations, etc.)
~~4. Determine if alerting features should be enabled, if so...~~
~~[] document guidelines for alerting functionality~~
5. Overall validation and cut over
[x] Provide access to new environment widely, with old env still available as a backup. (https://logstash-next.wikimedia.org)
[x] Gather/address bugs identified during this period
[x] Perform cut-over (name switch to logstash.wm.o)
6. Migrate Kafka-logging brokers to ELK 7 cluster
7. Fold (reimage/migrate) ELK 5 hardware into ELK7 cluster
8. Retire ELK 5 VMs