Page MenuHomePhabricator

Standardize the logging format
Open, MediumPublic

Description

A common logging format is desirable for a few reasons (list may not be complete):

  1. to apply an index schema and enable proper type coercion when logs are indexed in Elasticsearch (related incident)
  2. to simplify the log messages and make them easier to reason about and query for
  3. to speed up searches
  4. to simplify de-duplication efforts
  5. to simplify detection and removal of useless logs
  6. to make the upgrade to newer versions of Elasticsearch possible (Single-type mapping per index enforced -- 6.x | 7.x)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+5 -1
operations/puppetproduction+87 -1
operations/puppetproduction+286 -6
operations/puppetproduction+1 -1
operations/puppetproduction+64 -39
operations/puppetproduction+1 -1
operations/puppetproduction+35 -4
operations/puppetproduction+4 -105
operations/puppetproduction+293 -0
operations/puppetproduction+209 -0
operations/puppetproduction+10 -3
operations/puppetproduction+28 -69
operations/puppetproduction+190 -3
operations/puppetproduction+76 -53
operations/puppetproduction+57 -31
operations/puppetproduction+13 -0
operations/puppetproduction+611 -0
operations/puppetproduction+19 -46
integration/configmaster+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+21 -0
operations/puppetproduction+46 -1
operations/puppetproduction+119 -0
operations/puppetproduction+3 -1
operations/puppetproduction+107 -0
operations/puppetproduction+39 -90
operations/puppetproduction+211 -0
operations/puppetproduction+3 -1
operations/puppetproduction+115 -79
operations/puppetproduction+8 -9
operations/puppetproduction+11 -8
operations/puppetproduction+79 -0
operations/puppetproduction+75 -16
operations/puppetproduction+78 -0
operations/puppetproduction+39 -1
operations/puppetproduction+23 -16
operations/puppetproduction+23 -0
operations/puppetproduction+14 -0
operations/puppetproduction+113 -0
operations/puppetproduction+3 K -2
operations/puppetproduction+76 -0
operations/puppetproduction+25 -20
operations/puppetproduction+3 K -1
integration/docrootmaster+5 -0
integration/docrootmaster+5 -0
integration/configmaster+42 -0
integration/configmaster+43 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 657213 merged by Cwhite:
[operations/puppet@production] profile: drop ECS messages on legacy cluster

https://gerrit.wikimedia.org/r/657213

Change 657370 merged by Cwhite:
[operations/puppet@production] logstash: enable curator to accept custom age filters

https://gerrit.wikimedia.org/r/657370

Change 657371 merged by Cwhite:
[operations/puppet@production] profile: ecs indices to use a weekly rotation

https://gerrit.wikimedia.org/r/657371

Change 647028 merged by Cwhite:
[operations/puppet@production] profile: add ecs pre and post filters to pipeline

https://gerrit.wikimedia.org/r/647028

Change 659422 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: only set default partition if unset

https://gerrit.wikimedia.org/r/659422

Change 659426 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add scap log duplication and ecs mutations

https://gerrit.wikimedia.org/r/659426

Change 661539 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: onboard icinga alerts to common logging schema

https://gerrit.wikimedia.org/r/661539

Change 661539 merged by Cwhite:
[operations/puppet@production] profile: onboard icinga logging to common logging schema

https://gerrit.wikimedia.org/r/661539

Change 647029 merged by Cwhite:
[operations/puppet@production] profile: update netdev to output ECS-formatted logs

https://gerrit.wikimedia.org/r/647029

Change 647032 merged by Cwhite:
[operations/puppet@production] profile: update netdev rsyslog template to ecs 1.7.0

https://gerrit.wikimedia.org/r/647032

Change 663613 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: remove type field for all ecs-formatted events

https://gerrit.wikimedia.org/r/663613

Change 663613 merged by Cwhite:
[operations/puppet@production] profile: remove type field for all ecs-formatted events

https://gerrit.wikimedia.org/r/663613

Change 663876 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add gerrit log duplication and ecs mutations

https://gerrit.wikimedia.org/r/663876

Change 647265 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add ulogd ecs filter + tests

https://gerrit.wikimedia.org/r/647265

Mentioned in SAL (#wikimedia-operations) [2021-02-17T16:46:21Z] <godog> roll-restart logstash to apply ulogd filter - T234565

Mentioned in SAL (#wikimedia-operations) [2021-02-17T17:36:32Z] <godog> roll-restart logstash7 in codfw/eqiad to apply ulogd filters - T234565

Change 659422 merged by Cwhite:
[operations/puppet@production] profile: only set default partition if unset

https://gerrit.wikimedia.org/r/659422

Change 663876 merged by Cwhite:
[operations/puppet@production] profile: add gerrit log duplication and ecs mutations

https://gerrit.wikimedia.org/r/663876

Change 668109 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: swap gerrit log stream to be ecs-only

https://gerrit.wikimedia.org/r/668109

Change 668109 merged by Cwhite:
[operations/puppet@production] profile: swap gerrit log stream to be ecs-only

https://gerrit.wikimedia.org/r/668109

Change 668231 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] httpd: add wmfecsjson logformat to defaults.conf

https://gerrit.wikimedia.org/r/668231

Change 659426 merged by Cwhite:
[operations/puppet@production] profile: add scap log duplication and ecs mutations

https://gerrit.wikimedia.org/r/659426

Change 670525 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: extract index label from logEvent indexing errors

https://gerrit.wikimedia.org/r/670525

Change 670525 merged by Cwhite:
[operations/puppet@production] logstash: extract index label from logEvent indexing errors

https://gerrit.wikimedia.org/r/670525

Change 668231 merged by Cwhite:
[operations/puppet@production] httpd: enable httpd to emit ECS-compliant logs

https://gerrit.wikimedia.org/r/668231

Change 670950 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] phabricator: use ecs-compatible apache log format

https://gerrit.wikimedia.org/r/670950

Change 670951 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] gerrit: use ecs-compatible apache access log format

https://gerrit.wikimedia.org/r/670951

Change 670986 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: rename logEvent exception into error.message

https://gerrit.wikimedia.org/r/670986

Change 670991 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: grok field name out of error.message

https://gerrit.wikimedia.org/r/670991

Change 670986 merged by Cwhite:
[operations/puppet@production] logstash: rename logEvent exception into error.message

https://gerrit.wikimedia.org/r/670986

Change 670991 merged by Cwhite:
[operations/puppet@production] logstash: grok field name out of error.message

https://gerrit.wikimedia.org/r/670991

Change 670951 merged by Dzahn:
[operations/puppet@production] gerrit: use ecs-compatible apache access log format

https://gerrit.wikimedia.org/r/670951

Change 670950 merged by Dzahn:
[operations/puppet@production] phabricator: use ecs-compatible apache access log format

https://gerrit.wikimedia.org/r/670950

Gerrit and Phabricator now have Apache 2 access log ingested. I have created a saved search in Kibana for source.type: apache2, added a couple very basic visualizations for top source.ip and top domain.name and crafted a dashboard:

https://logstash.wikimedia.org/app/dashboards#/view/825c5c80-8aef-11eb-8ab2-63c7f3b019fc

I have linked it on the main dashboard page as Apache (ECS)

Change 674718 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: replace ECS allow list with filter_on_template

https://gerrit.wikimedia.org/r/674718

Change 674720 had a related patch set uploaded (by Cwhite; author: Cwhite):
[integration/config@master] mount templates directory for logstash-filter-verifier

https://gerrit.wikimedia.org/r/674720

Change 674720 merged by jenkins-bot:
[integration/config@master] mount templates directory for logstash-filter-verifier

https://gerrit.wikimedia.org/r/674720

Change 685987 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: turn off scap duplication

https://gerrit.wikimedia.org/r/685987

Change 685987 merged by Cwhite:

[operations/puppet@production] profile: turn off scap duplication

https://gerrit.wikimedia.org/r/685987

Change 688502 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] rsyslog: add ecs_170 template

https://gerrit.wikimedia.org/r/688502

Change 689160 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] rsyslog: enable ecs_170 template and transition prometheus

https://gerrit.wikimedia.org/r/689160

Change 689262 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add openstack transition config and tests

https://gerrit.wikimedia.org/r/689262

Change 690759 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add nodejs ecs migration config and tests

https://gerrit.wikimedia.org/r/690759

Change 690759 merged by Cwhite:

[operations/puppet@production] logstash: add nodejs ecs migration config and tests

https://gerrit.wikimedia.org/r/690759

Change 688502 merged by Cwhite:

[operations/puppet@production] rsyslog: add ecs_170 template

https://gerrit.wikimedia.org/r/688502

Change 689160 merged by Cwhite:

[operations/puppet@production] rsyslog: enable ecs_170 template and transition prometheus

https://gerrit.wikimedia.org/r/689160

Change 695563 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add ECS transition support for Oslo structured logs

https://gerrit.wikimedia.org/r/695563

Change 674718 merged by Cwhite:

[operations/puppet@production] logstash: replace ECS allow list with filter_on_templates

https://gerrit.wikimedia.org/r/674718

Change 689262 merged by Cwhite:

[operations/puppet@production] logstash: add openstack ECS transition config and tests

https://gerrit.wikimedia.org/r/689262

Change 699039 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: transition openstack to ECS

https://gerrit.wikimedia.org/r/699039

Change 699254 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add ecs migration config for sampled webrequest logs

https://gerrit.wikimedia.org/r/699254

Change 699039 merged by Cwhite:

[operations/puppet@production] logstash: transition openstack to ECS

https://gerrit.wikimedia.org/r/699039

Change 701617 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: transition aqs logs to ECS

https://gerrit.wikimedia.org/r/701617

Change 701617 merged by Cwhite:

[operations/puppet@production] logstash: transition aqs logs to ECS

https://gerrit.wikimedia.org/r/701617

Change 699254 merged by Cwhite:

[operations/puppet@production] logstash: add ecs migration config for sampled webrequest logs

https://gerrit.wikimedia.org/r/699254

Change 695563 merged by Cwhite:

[operations/puppet@production] logstash: add ECS transition support for Oslo structured logs

https://gerrit.wikimedia.org/r/695563

Change 705729 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: complete restbase transition to ECS

https://gerrit.wikimedia.org/r/705729

Change 705729 merged by Cwhite:

[operations/puppet@production] logstash: complete restbase transition to ECS

https://gerrit.wikimedia.org/r/705729

I'm slightly confused by the state of the filters for openstack/oslo. Right now I see:

15-filter_oslo_json.conf
15-filter_openstack_migration.conf
50-filter_openstack.conf

I would like to add a new filter for nova-fullstack (https://gerrit.wikimedia.org/r/c/operations/puppet/+/713006) but am unsure where to put it. Can we drop 50-filter_openstack.conf entirely? And/or merge the other two files?

Change 713314 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] openstack: adapt nova_fullstack_test to emit ECS-compatibile logs

https://gerrit.wikimedia.org/r/713314

I'm slightly confused by the state of the filters for openstack/oslo.

50-filter_openstack.conf is most likely unused at this point and could probably be removed.

I would like to add a new filter for nova-fullstack (https://gerrit.wikimedia.org/r/c/operations/puppet/+/713006) but am unsure where to put it. Can we drop 50-filter_openstack.conf entirely? And/or merge the other two files?

15-filter_oslo_json.conf is set up to transform oslo json-formatted logs (only Trove right now, IIRC). Once the other Openstack components are configured to emit oslo json-formatted logs, 15-filter_openstack_migration.conf should eventually be removed.

For nova_fullstack_test, I think it's worth attempting to get it to emit ECS-compatbile logs. See https://gerrit.wikimedia.org/r/713314 and let me know what you think!

Change 713314 merged by Cwhite:

[operations/puppet@production] openstack: adapt nova_fullstack_test to emit ECS-compatibile logs

https://gerrit.wikimedia.org/r/713314

Change 713318 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] openstack: cast record.msg to string in the formatter

https://gerrit.wikimedia.org/r/713318

Change 713318 merged by Andrew Bogott:

[operations/puppet@production] openstack: cast record.msg to string in the formatter

https://gerrit.wikimedia.org/r/713318

Change 713323 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: forward nova-fullstack logs to logstash

https://gerrit.wikimedia.org/r/713323

Change 713323 merged by Andrew Bogott:

[operations/puppet@production] logstash: forward nova-fullstack logs to logstash

https://gerrit.wikimedia.org/r/713323

Change 717489 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: route aqs and restbase logs to default ecs indexes

https://gerrit.wikimedia.org/r/717489

Change 717489 merged by Cwhite:

[operations/puppet@production] logstash: route aqs and restbase logs to default ecs indexes

https://gerrit.wikimedia.org/r/717489

Change 730897 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: duplicate MediaWiki error,fatal,exception logs to ECS test

https://gerrit.wikimedia.org/r/730897

Change 730897 merged by Cwhite:

[operations/puppet@production] logstash: duplicate MediaWiki error and exception logs to ECS test

https://gerrit.wikimedia.org/r/730897

Change 803947 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add php7.2-fpm to mediawiki error,exception processing

https://gerrit.wikimedia.org/r/803947

Change 803947 merged by Cwhite:

[operations/puppet@production] logstash: add php7.2-fpm to mediawiki error,exception processing

https://gerrit.wikimedia.org/r/803947

Change 804010 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: truncate labels.normalized_message

https://gerrit.wikimedia.org/r/804010

Change 804010 merged by Cwhite:

[operations/puppet@production] logstash: truncate labels.normalized_message

https://gerrit.wikimedia.org/r/804010