Tracking task for upgrading the ELK stack to a more current stable release (targeting version 7.2)
High level items
- Build an ELK 7 upgrade environment in parallel to production
- Provision ES 7 hosts (HW & OS)
- Provision Logstash/Kibana 7 collector hosts (VM & OS)
- Make new versions of ELK software installable via apt
- Puppetize logging ES 7
- Puppetize Logstash 7
- Puppetize Kibana 7
- Configure service address for load balanced Kibana frontend
2. Determine legal viability of amazon open distro for elasticsearch, if so
[] Integrate RBAC features with LDAP
[] Puppetize management of security users, roles, mappings, etc.
- Ingest production logs
- Determine best way to handle/manage logstash plugins in the new version & execute
- Consume from kafka-logging
- Determine best method to bridge gap for ingesting log sources not not yet in Kafka
- Validate log parsing, storage, etc.
- Investigate and upgrade/adapt curator as necessary
- Import Kibana configuration (saved searches, dashboards, visualizations, etc.)
4. Determine if alerting features should be enabled, if so...
[] document guidelines for alerting functionality
- Overall validation and cut over
- Provide access to new environment widely, with old env still available as a backup. (https://logstash-next.wikimedia.org)
- Gather/address bugs identified during this period
- Perform cut-over (name switch to logstash.wm.o)
- Provide access to new environment widely, with old env still available as a backup. (https://logstash-next.wikimedia.org)
- Migrate Kafka-logging brokers to ELK 7 cluster
- Fold (reimage/migrate) ELK 5 hardware into ELK7 cluster
- Retire ELK 5 VMs