We have more and more OpenStack boxes; it would be great to get their service logs forwarded to kibana so I don't have to log into 30 hosts to troubleshoot.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T255990 Openstack: consider introducing log aggregation | |||
| Resolved | Andrew | T268175 central logging for OpenStack services | |||
| Resolved | Andrew | T268176 Allow kafka network access to cloudvirt hosts | |||
| Resolved | Andrew | T272117 Clean up deprecation warnings in OpenStack logs |
Change 641830 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Nova: Pass some logs into our central ELK stack
Change 641830 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Nova: Pass some logs into our central ELK stack
Change 642025 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add log filter for Openstack/Oslo log messages
There's an upstream catch-all for filtering openstack related services:
https://github.com/openstack/logstash-filters
That relies heavily on the 'multiline' plugin; we may want to add that at some point.
Change 642025 merged by Andrew Bogott:
[operations/puppet@production] Add log filter for Openstack/Oslo log messages
Change 642078 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add test for OpenStack/Osl log filter; update filter to cooperate
Change 642917 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Designate: pass logs to central ELK stack
Change 642078 merged by Cwhite:
[operations/puppet@production] Add test for OpenStack/Oslo log filter; update filter to cooperate
Change 642917 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Designate: pass logs to central ELK stack
Change 643074 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack log filters: improve handling of req-id
Change 643087 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack: direct neutron, glance, keystone logs to logstash
Change 643087 merged by Andrew Bogott:
[operations/puppet@production] OpenStack: direct neutron, glance, keystone logs to logstash
Change 643074 merged by Cwhite:
[operations/puppet@production] Openstack log filters: improve handling of req-id
Change 643351 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Glance: filter out health check log messages
Change 643351 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Glance: filter out health check log messages
Change 643353 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack log filters: don't process Glance logs
Change 643353 merged by Andrew Bogott:
[operations/puppet@production] Openstack log filters: don't process Glance logs
Change 643358 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Keystone: remove use of log_config_append
Change 643358 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Keystone: remove use of log_config_append
Change 643364 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Rename keystone programs for log filters
Change 643364 merged by Andrew Bogott:
[operations/puppet@production] Rename keystone programs for log filters
I've been using local3 for network related logs since a few years now as it was unused. Unfortunately, this wasn't documented.
I was wondering of you could refrain from using that facility until T268806 is solved?
Change 643995 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack central logging: use LOG_LOCAL0 for everything
Change 643995 merged by Andrew Bogott:
[operations/puppet@production] OpenStack central logging: use LOG_LOCAL0 for everything
Change 644063 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: turn off INFO-level logging
Change 644063 merged by Andrew Bogott:
[operations/puppet@production] Keystone: turn off INFO-level logging
Change 644064 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: set log levels to recommended upstream defaults
Change 644064 merged by Andrew Bogott:
[operations/puppet@production] designate: set log levels to recommended upstream defaults
Change 644066 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack Glance: further attempt to quiet down logging a bit
Change 644066 merged by Andrew Bogott:
[operations/puppet@production] OpenStack Glance: further attempt to quiet down logging a bit
Change 647850 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Add cinder logs to central logging
Change 647850 merged by Andrew Bogott:
[operations/puppet@production] Add cinder logs to central logging
Change 648296 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder: Fix syslog filter of health checks
Change 648296 merged by Andrew Bogott:
[operations/puppet@production] Cinder: Fix syslog filter of health checks
Change 648302 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Cinder policy.yaml: fix typo that broke policy parsing
Change 648302 merged by Andrew Bogott:
[operations/puppet@production] Cinder policy.yaml: fix typo that broke policy parsing
Change 651186 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack haproxy: logs to rsyslog
Change 651187 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] OpenStack haproxy: direct logs to ELK
Change 651187 merged by Andrew Bogott:
[operations/puppet@production] OpenStack haproxy: direct logs to ELK
Change 651252 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Direct more OpenStack logs to kafka/kibana
Change 651252 merged by Andrew Bogott:
[operations/puppet@production] Direct more OpenStack logs to kafka/kibana
Change 651186 abandoned by Andrew Bogott:
[operations/puppet@production] OpenStack haproxy: logs to rsyslog
Reason:
incorrect!
Change 651254 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: filter health checks from Kibana logs
Change 651254 merged by Andrew Bogott:
[operations/puppet@production] Designate: filter health checks from Kibana logs
Change 651256 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron: change use_syslog from True to true
Change 651256 merged by Andrew Bogott:
[operations/puppet@production] Neutron: change use_syslog from True to true
Change 651259 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Designate: further syslog filters for designate-api
Change 651259 merged by Andrew Bogott:
[operations/puppet@production] Designate: further syslog filters for designate-api
Change 664678 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack control node galera: send mariadb logs to central logging
Change 664678 abandoned by Andrew Bogott:
[operations/puppet@production] Openstack control node galera: send mariadb logs to central logging
Reason:
Jaime isn't sure that we can safely send mariadb logs centrally and I was nervous about that already, so I'm leaving those stranded on the galera nodes for now.
Change 664934 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: ship error logs to ELK
Change 664934 merged by Andrew Bogott:
[operations/puppet@production] Horizon: ship error logs to ELK
Change 664943 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] lookup_table_output.json: Send horizon logs to kafka
Change 664943 merged by Andrew Bogott:
[operations/puppet@production] lookup_table_output.json: Send horizon logs to kafka
Change 665164 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon logging: try to detect multiple-line log messages
Change 665164 merged by Andrew Bogott:
[operations/puppet@production] Horizon logging: try to detect multiple-line log messages
I've now created dashboards for horizon:
https://logstash.wikimedia.org/app/dashboards#/view/71c74950-716a-11eb-ba38-efc6fae89eca (eqiad1)
and
https://logstash.wikimedia.org/app/dashboards#/view/fdf3b3b0-7227-11eb-8327-370b46f9e7a5 (codfw1dev)