Page MenuHomePhabricator

Standardize the logging format
Open, MediumPublic

Description

A common logging format is desirable for a few reasons (list may not be complete):

  1. to apply an index schema and enable proper type coercion when logs are indexed in Elasticsearch (related incident)
  2. to simplify the log messages and make them easier to reason about and query for
  3. to speed up searches
  4. to simplify de-duplication efforts
  5. to simplify detection and removal of useless logs
  6. to make the upgrade to newer versions of Elasticsearch possible (Single-type mapping per index enforced -- 6.x | 7.x)

Details

SubjectRepoBranchLines +/-
operations/puppetproduction+114 -5
operations/puppetproduction+57 -95
operations/puppetproduction+6 -22
operations/puppetproduction+16 -33
operations/puppetproduction+21 -58
operations/puppetproduction+27 -64
operations/puppetproduction+6 -21
operations/puppetproduction+1 -1
operations/puppetproduction+72 -0
operations/puppetproduction+94 -0
operations/puppetproduction+213 -0
operations/puppetproduction+186 -0
operations/puppetproduction+71 -0
operations/puppetproduction+25 -4
operations/puppetproduction+3 -0
operations/puppetproduction+21 -5
operations/puppetproduction+6 -0
operations/puppetproduction+5 -1
operations/puppetproduction+87 -1
operations/puppetproduction+286 -6
operations/puppetproduction+1 -1
operations/puppetproduction+64 -39
operations/puppetproduction+1 -1
operations/puppetproduction+35 -4
operations/puppetproduction+4 -105
operations/puppetproduction+293 -0
operations/puppetproduction+209 -0
operations/puppetproduction+10 -3
operations/puppetproduction+28 -69
operations/puppetproduction+190 -3
operations/puppetproduction+76 -53
operations/puppetproduction+57 -31
operations/puppetproduction+13 -0
operations/puppetproduction+611 -0
operations/puppetproduction+19 -46
integration/configmaster+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+21 -0
operations/puppetproduction+46 -1
operations/puppetproduction+119 -0
operations/puppetproduction+3 -1
operations/puppetproduction+107 -0
operations/puppetproduction+39 -90
operations/puppetproduction+211 -0
operations/puppetproduction+3 -1
operations/puppetproduction+115 -79
operations/puppetproduction+8 -9
operations/puppetproduction+11 -8
operations/puppetproduction+79 -0
operations/puppetproduction+75 -16
operations/puppetproduction+78 -0
operations/puppetproduction+39 -1
operations/puppetproduction+23 -16
operations/puppetproduction+23 -0
operations/puppetproduction+14 -0
operations/puppetproduction+113 -0
operations/puppetproduction+3 K -2
operations/puppetproduction+76 -0
operations/puppetproduction+25 -20
operations/puppetproduction+3 K -1
integration/docrootmaster+5 -0
integration/docrootmaster+5 -0
integration/configmaster+42 -0
integration/configmaster+43 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 690759 merged by Cwhite:

[operations/puppet@production] logstash: add nodejs ecs migration config and tests

https://gerrit.wikimedia.org/r/690759

Change 688502 merged by Cwhite:

[operations/puppet@production] rsyslog: add ecs_170 template

https://gerrit.wikimedia.org/r/688502

Change 689160 merged by Cwhite:

[operations/puppet@production] rsyslog: enable ecs_170 template and transition prometheus

https://gerrit.wikimedia.org/r/689160

Change 695563 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add ECS transition support for Oslo structured logs

https://gerrit.wikimedia.org/r/695563

Change 674718 merged by Cwhite:

[operations/puppet@production] logstash: replace ECS allow list with filter_on_templates

https://gerrit.wikimedia.org/r/674718

Change 689262 merged by Cwhite:

[operations/puppet@production] logstash: add openstack ECS transition config and tests

https://gerrit.wikimedia.org/r/689262

Change 699039 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: transition openstack to ECS

https://gerrit.wikimedia.org/r/699039

Change 699254 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add ecs migration config for sampled webrequest logs

https://gerrit.wikimedia.org/r/699254

Change 699039 merged by Cwhite:

[operations/puppet@production] logstash: transition openstack to ECS

https://gerrit.wikimedia.org/r/699039

Change 701617 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: transition aqs logs to ECS

https://gerrit.wikimedia.org/r/701617

Change 701617 merged by Cwhite:

[operations/puppet@production] logstash: transition aqs logs to ECS

https://gerrit.wikimedia.org/r/701617

Change 699254 merged by Cwhite:

[operations/puppet@production] logstash: add ecs migration config for sampled webrequest logs

https://gerrit.wikimedia.org/r/699254

Change 695563 merged by Cwhite:

[operations/puppet@production] logstash: add ECS transition support for Oslo structured logs

https://gerrit.wikimedia.org/r/695563

Change 705729 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: complete restbase transition to ECS

https://gerrit.wikimedia.org/r/705729

Change 705729 merged by Cwhite:

[operations/puppet@production] logstash: complete restbase transition to ECS

https://gerrit.wikimedia.org/r/705729

I'm slightly confused by the state of the filters for openstack/oslo. Right now I see:

15-filter_oslo_json.conf
15-filter_openstack_migration.conf
50-filter_openstack.conf

I would like to add a new filter for nova-fullstack (https://gerrit.wikimedia.org/r/c/operations/puppet/+/713006) but am unsure where to put it. Can we drop 50-filter_openstack.conf entirely? And/or merge the other two files?

Change 713314 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] openstack: adapt nova_fullstack_test to emit ECS-compatibile logs

https://gerrit.wikimedia.org/r/713314

I'm slightly confused by the state of the filters for openstack/oslo.

50-filter_openstack.conf is most likely unused at this point and could probably be removed.

I would like to add a new filter for nova-fullstack (https://gerrit.wikimedia.org/r/c/operations/puppet/+/713006) but am unsure where to put it. Can we drop 50-filter_openstack.conf entirely? And/or merge the other two files?

15-filter_oslo_json.conf is set up to transform oslo json-formatted logs (only Trove right now, IIRC). Once the other Openstack components are configured to emit oslo json-formatted logs, 15-filter_openstack_migration.conf should eventually be removed.

For nova_fullstack_test, I think it's worth attempting to get it to emit ECS-compatbile logs. See https://gerrit.wikimedia.org/r/713314 and let me know what you think!

Change 713314 merged by Cwhite:

[operations/puppet@production] openstack: adapt nova_fullstack_test to emit ECS-compatibile logs

https://gerrit.wikimedia.org/r/713314

Change 713318 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] openstack: cast record.msg to string in the formatter

https://gerrit.wikimedia.org/r/713318

Change 713318 merged by Andrew Bogott:

[operations/puppet@production] openstack: cast record.msg to string in the formatter

https://gerrit.wikimedia.org/r/713318

Change 713323 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: forward nova-fullstack logs to logstash

https://gerrit.wikimedia.org/r/713323

Change 713323 merged by Andrew Bogott:

[operations/puppet@production] logstash: forward nova-fullstack logs to logstash

https://gerrit.wikimedia.org/r/713323

Change 717489 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: route aqs and restbase logs to default ecs indexes

https://gerrit.wikimedia.org/r/717489

Change 717489 merged by Cwhite:

[operations/puppet@production] logstash: route aqs and restbase logs to default ecs indexes

https://gerrit.wikimedia.org/r/717489

Change 730897 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: duplicate MediaWiki error,fatal,exception logs to ECS test

https://gerrit.wikimedia.org/r/730897

Change 730897 merged by Cwhite:

[operations/puppet@production] logstash: duplicate MediaWiki error and exception logs to ECS test

https://gerrit.wikimedia.org/r/730897

Change 803947 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add php7.2-fpm to mediawiki error,exception processing

https://gerrit.wikimedia.org/r/803947

Change 803947 merged by Cwhite:

[operations/puppet@production] logstash: add php7.2-fpm to mediawiki error,exception processing

https://gerrit.wikimedia.org/r/803947

Change 804010 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: truncate labels.normalized_message

https://gerrit.wikimedia.org/r/804010

Change 804010 merged by Cwhite:

[operations/puppet@production] logstash: truncate labels.normalized_message

https://gerrit.wikimedia.org/r/804010

colewhite claimed this task.

Change 895741 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move mediawiki ecs logs into mediawiki partition

https://gerrit.wikimedia.org/r/895741

Change 895739 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: mediawiki_ecs copy http_method into place

https://gerrit.wikimedia.org/r/895739

Change 895741 merged by Cwhite:

[operations/puppet@production] logstash: move mediawiki ecs logs into mediawiki partition

https://gerrit.wikimedia.org/r/895741

Change 900719 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add tag on json parsing log field

https://gerrit.wikimedia.org/r/900719

Change 900719 merged by Cwhite:

[operations/puppet@production] logstash: add tag on json parsing log field

https://gerrit.wikimedia.org/r/900719

Change 901630 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add mmkubernetes ECS early-stage filter

https://gerrit.wikimedia.org/r/901630

Change 901631 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add k8s statsd-exporter ECS filters and tests

https://gerrit.wikimedia.org/r/901631

Change 902081 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] logstash: add pybal ECS filter and tests

https://gerrit.wikimedia.org/r/902081

Change 901630 merged by Cwhite:

[operations/puppet@production] logstash: add mmkubernetes ECS early-stage filter

https://gerrit.wikimedia.org/r/901630

Change 901642 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add grafana-server ECS filters and tests

https://gerrit.wikimedia.org/r/901642

Change 902081 merged by Cwhite:

[operations/puppet@production] logstash: add pybal ECS filter and tests

https://gerrit.wikimedia.org/r/902081

Change 902334 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add thanos-query ECS filters and tests

https://gerrit.wikimedia.org/r/902334

Change 902611 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] add haproxy ECS filters and tests

https://gerrit.wikimedia.org/r/902611

Change 901642 merged by Cwhite:

[operations/puppet@production] logstash: add grafana-server ECS filters and tests

https://gerrit.wikimedia.org/r/901642

Change 901631 merged by Cwhite:

[operations/puppet@production] logstash: add k8s statsd-exporter ECS filters and tests

https://gerrit.wikimedia.org/r/901631

Change 902334 merged by Cwhite:

[operations/puppet@production] logstash: add thanos-query ECS filters and tests

https://gerrit.wikimedia.org/r/902334

Change 902611 merged by Cwhite:

[operations/puppet@production] add haproxy ECS filters and tests

https://gerrit.wikimedia.org/r/902611

Change 937600 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove pybal log cloning

https://gerrit.wikimedia.org/r/937600

Change 937601 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove haproxy log cloning

https://gerrit.wikimedia.org/r/937601

Change 937602 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove grafana log cloning

https://gerrit.wikimedia.org/r/937602

Change 937603 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove k8s stats-exporter cloning

https://gerrit.wikimedia.org/r/937603

Change 937604 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove thanos log cloning

https://gerrit.wikimedia.org/r/937604

Change 937605 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: restore program field to node logs

https://gerrit.wikimedia.org/r/937605

Change 938326 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove node log cloning

https://gerrit.wikimedia.org/r/938326

Change 937605 merged by Cwhite:

[operations/puppet@production] logstash: restore program field to node logs

https://gerrit.wikimedia.org/r/937605

Change 937600 merged by Cwhite:

[operations/puppet@production] logstash: remove pybal log cloning

https://gerrit.wikimedia.org/r/937600

Change 937602 merged by Cwhite:

[operations/puppet@production] logstash: remove grafana log cloning

https://gerrit.wikimedia.org/r/937602

Change 937603 merged by Cwhite:

[operations/puppet@production] logstash: remove k8s stats-exporter cloning

https://gerrit.wikimedia.org/r/937603

Change 937604 merged by Cwhite:

[operations/puppet@production] logstash: remove thanos log cloning

https://gerrit.wikimedia.org/r/937604

Change 937601 merged by Cwhite:

[operations/puppet@production] logstash: remove haproxy log cloning

https://gerrit.wikimedia.org/r/937601

Change 938326 merged by Cwhite:

[operations/puppet@production] logstash: remove node log cloning

https://gerrit.wikimedia.org/r/938326

Change 951881 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move error to error.message when it is a string

https://gerrit.wikimedia.org/r/951881