Page MenuHomePhabricator

Standardize the logging format
Open, MediumPublic

Description

A common logging format is desirable for a few reasons (list may not be complete):

  1. to apply an index schema and enable proper type coercion when logs are indexed in Elasticsearch (related incident)
  2. to simplify the log messages and make them easier to reason about and query for
  3. to speed up searches
  4. to simplify de-duplication efforts
  5. to simplify detection and removal of useless logs
  6. to make the upgrade to newer versions of Elasticsearch possible (Single-type mapping per index enforced -- 6.x | 7.x)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+114 -2
operations/puppetproduction+3 -3
operations/puppetproduction+0 -51
operations/software/ecsmaster+9 -0
operations/puppetproduction+13 -22
operations/puppetproduction+2 -49
operations/puppetproduction+55 -2
operations/puppetproduction+13 -122
operations/puppetproduction+2 -1
operations/puppetproduction+24 -9
operations/puppetproduction+23 -0
operations/puppetproduction+263 -42
operations/puppetproduction+11 -2
operations/puppetproduction+1 -1
operations/puppetproduction+48 -26
operations/software/ecsmaster+12 -5
operations/software/ecsmaster+7 -0
operations/software/ecsmaster+3 -0
operations/puppetproduction+20 -0
operations/puppetproduction+476 -0
operations/puppetproduction+193 -44
operations/puppetproduction+57 -95
operations/puppetproduction+6 -22
operations/puppetproduction+16 -33
operations/puppetproduction+21 -58
operations/puppetproduction+27 -64
operations/puppetproduction+6 -21
operations/puppetproduction+1 -1
operations/puppetproduction+72 -0
operations/puppetproduction+94 -0
operations/puppetproduction+213 -0
operations/puppetproduction+186 -0
operations/puppetproduction+71 -0
operations/puppetproduction+25 -4
operations/puppetproduction+3 -0
operations/puppetproduction+21 -5
operations/puppetproduction+6 -0
operations/puppetproduction+5 -1
operations/puppetproduction+87 -1
operations/puppetproduction+286 -6
operations/puppetproduction+1 -1
operations/puppetproduction+64 -39
operations/puppetproduction+1 -1
operations/puppetproduction+35 -4
operations/puppetproduction+4 -105
operations/puppetproduction+293 -0
operations/puppetproduction+209 -0
operations/puppetproduction+10 -3
operations/puppetproduction+28 -69
operations/puppetproduction+190 -3
operations/puppetproduction+76 -53
operations/puppetproduction+57 -31
operations/puppetproduction+13 -0
operations/puppetproduction+611 -0
operations/puppetproduction+19 -46
integration/configmaster+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+21 -0
operations/puppetproduction+46 -1
operations/puppetproduction+119 -0
operations/puppetproduction+3 -1
operations/puppetproduction+107 -0
operations/puppetproduction+39 -90
operations/puppetproduction+211 -0
operations/puppetproduction+3 -1
operations/puppetproduction+115 -79
operations/puppetproduction+8 -9
operations/puppetproduction+11 -8
operations/puppetproduction+79 -0
operations/puppetproduction+75 -16
operations/puppetproduction+78 -0
operations/puppetproduction+39 -1
operations/puppetproduction+23 -16
operations/puppetproduction+23 -0
operations/puppetproduction+14 -0
operations/puppetproduction+113 -0
operations/puppetproduction+3 K -2
operations/puppetproduction+76 -0
operations/puppetproduction+25 -20
operations/puppetproduction+3 K -1
integration/docrootmaster+5 -0
integration/docrootmaster+5 -0
integration/configmaster+42 -0
integration/configmaster+43 -0
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
colewhite claimed this task.

Change 895741 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move mediawiki ecs logs into mediawiki partition

https://gerrit.wikimedia.org/r/895741

Change 895739 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: mediawiki_ecs copy http_method into place

https://gerrit.wikimedia.org/r/895739

Change 895741 merged by Cwhite:

[operations/puppet@production] logstash: move mediawiki ecs logs into mediawiki partition

https://gerrit.wikimedia.org/r/895741

Change 900719 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add tag on json parsing log field

https://gerrit.wikimedia.org/r/900719

Change 900719 merged by Cwhite:

[operations/puppet@production] logstash: add tag on json parsing log field

https://gerrit.wikimedia.org/r/900719

Change 901630 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add mmkubernetes ECS early-stage filter

https://gerrit.wikimedia.org/r/901630

Change 901631 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add k8s statsd-exporter ECS filters and tests

https://gerrit.wikimedia.org/r/901631

Change 902081 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] logstash: add pybal ECS filter and tests

https://gerrit.wikimedia.org/r/902081

Change 901630 merged by Cwhite:

[operations/puppet@production] logstash: add mmkubernetes ECS early-stage filter

https://gerrit.wikimedia.org/r/901630

Change 901642 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add grafana-server ECS filters and tests

https://gerrit.wikimedia.org/r/901642

Change 902081 merged by Cwhite:

[operations/puppet@production] logstash: add pybal ECS filter and tests

https://gerrit.wikimedia.org/r/902081

Change 902334 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add thanos-query ECS filters and tests

https://gerrit.wikimedia.org/r/902334

Change 902611 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] add haproxy ECS filters and tests

https://gerrit.wikimedia.org/r/902611

Change 901642 merged by Cwhite:

[operations/puppet@production] logstash: add grafana-server ECS filters and tests

https://gerrit.wikimedia.org/r/901642

Change 901631 merged by Cwhite:

[operations/puppet@production] logstash: add k8s statsd-exporter ECS filters and tests

https://gerrit.wikimedia.org/r/901631

Change 902334 merged by Cwhite:

[operations/puppet@production] logstash: add thanos-query ECS filters and tests

https://gerrit.wikimedia.org/r/902334

Change 902611 merged by Cwhite:

[operations/puppet@production] add haproxy ECS filters and tests

https://gerrit.wikimedia.org/r/902611

Change 937600 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove pybal log cloning

https://gerrit.wikimedia.org/r/937600

Change 937601 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove haproxy log cloning

https://gerrit.wikimedia.org/r/937601

Change 937602 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove grafana log cloning

https://gerrit.wikimedia.org/r/937602

Change 937603 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove k8s stats-exporter cloning

https://gerrit.wikimedia.org/r/937603

Change 937604 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove thanos log cloning

https://gerrit.wikimedia.org/r/937604

Change 937605 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: restore program field to node logs

https://gerrit.wikimedia.org/r/937605

Change 938326 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove node log cloning

https://gerrit.wikimedia.org/r/938326

Change 937605 merged by Cwhite:

[operations/puppet@production] logstash: restore program field to node logs

https://gerrit.wikimedia.org/r/937605

Change 937600 merged by Cwhite:

[operations/puppet@production] logstash: remove pybal log cloning

https://gerrit.wikimedia.org/r/937600

Change 937602 merged by Cwhite:

[operations/puppet@production] logstash: remove grafana log cloning

https://gerrit.wikimedia.org/r/937602

Change 937603 merged by Cwhite:

[operations/puppet@production] logstash: remove k8s stats-exporter cloning

https://gerrit.wikimedia.org/r/937603

Change 937604 merged by Cwhite:

[operations/puppet@production] logstash: remove thanos log cloning

https://gerrit.wikimedia.org/r/937604

Change 937601 merged by Cwhite:

[operations/puppet@production] logstash: remove haproxy log cloning

https://gerrit.wikimedia.org/r/937601

Change 938326 merged by Cwhite:

[operations/puppet@production] logstash: remove node log cloning

https://gerrit.wikimedia.org/r/938326

Change 951881 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move error to error.message when it is a string

https://gerrit.wikimedia.org/r/951881

Change #1152850 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add early-stage filter to populate event.original

https://gerrit.wikimedia.org/r/1152850

Change #1154348 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add filter_on_template_v2

https://gerrit.wikimedia.org/r/1154348

Change #1152850 merged by Cwhite:

[operations/puppet@production] logstash: add early-stage filter to populate event.original

https://gerrit.wikimedia.org/r/1152850

Change #1163386 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: enable filter_ecs_cleanup_v2 on beta-logs

https://gerrit.wikimedia.org/r/1163386

Change #1154348 merged by Cwhite:

[operations/puppet@production] logstash: add filter_on_template_v2

https://gerrit.wikimedia.org/r/1154348

Change #1163386 merged by Cwhite:

[operations/puppet@production] logstash: enable filter_ecs_cleanup_v2 on beta-logs

https://gerrit.wikimedia.org/r/1163386

Change #1163486 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: explicitly define allowed numeric types

https://gerrit.wikimedia.org/r/1163486

Change #1163489 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: temporarily remove filter_on_template_v2 from beta-logs

https://gerrit.wikimedia.org/r/1163489

Change #1163489 merged by Cwhite:

[operations/puppet@production] logstash: temporarily remove filter_on_template_v2 from beta-logs

https://gerrit.wikimedia.org/r/1163489

Change #1164231 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/software/ecs@master] validator: amend get_type to handle nulls

https://gerrit.wikimedia.org/r/1164231

Change #1164231 merged by jenkins-bot:

[operations/software/ecs@master] validator: amend get_type to handle nulls

https://gerrit.wikimedia.org/r/1164231

Change #1164245 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/software/ecs@master] validator: keyword type can be an array of strings

https://gerrit.wikimedia.org/r/1164245

Change #1164251 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/software/ecs@master] validator: fix get_type to correctly handle ints

https://gerrit.wikimedia.org/r/1164251

Change #1164245 merged by jenkins-bot:

[operations/software/ecs@master] validator: keyword type can be an array of strings

https://gerrit.wikimedia.org/r/1164245

Change #1164251 merged by jenkins-bot:

[operations/software/ecs@master] validator: fix get_type to correctly handle ints

https://gerrit.wikimedia.org/r/1164251

Change #1164522 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: convert ECS numeric fields from strings

https://gerrit.wikimedia.org/r/1164522

Change #1164523 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: scale gitlab durations to expected unit (ns)

https://gerrit.wikimedia.org/r/1164523

Change #1164524 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move grafana status_code field to the right place

https://gerrit.wikimedia.org/r/1164524

Change #1164525 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: flatten array of objects in stack_trace

https://gerrit.wikimedia.org/r/1164525

Change #1164526 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: use filter_on_templates_v2

https://gerrit.wikimedia.org/r/1164526

Change #1163486 merged by Cwhite:

[operations/puppet@production] logstash: filter_on_template_v2 fixes

https://gerrit.wikimedia.org/r/1163486

Change #1165586 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: test filter_on_template_v2 on codfw

https://gerrit.wikimedia.org/r/1165586

Change #1165988 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: pass through normalized arrays from filter-on-template v1

https://gerrit.wikimedia.org/r/1165988

Change #1165988 merged by Cwhite:

[operations/puppet@production] logstash: pass through normalized arrays from filter-on-template v1

https://gerrit.wikimedia.org/r/1165988

Change #1166008 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/software/ecs@master] add docs for string_to_numeric_conversion_failure

https://gerrit.wikimedia.org/r/1166008

Change #1164522 merged by Cwhite:

[operations/puppet@production] logstash: convert ECS numeric fields from strings

https://gerrit.wikimedia.org/r/1164522

Change #1165586 merged by Cwhite:

[operations/puppet@production] logstash: test filter_on_template_v2 on codfw

https://gerrit.wikimedia.org/r/1165586

Change #1164523 merged by Cwhite:

[operations/puppet@production] logstash: scale gitlab durations to expected unit (ns)

https://gerrit.wikimedia.org/r/1164523

Change #1164524 merged by Cwhite:

[operations/puppet@production] logstash: move grafana status_code field to the right place

https://gerrit.wikimedia.org/r/1164524

Change #1164525 merged by Cwhite:

[operations/puppet@production] logstash: flatten array of objects in stack_trace

https://gerrit.wikimedia.org/r/1164525

Change #1167926 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: fix gitlab event field type conflict

https://gerrit.wikimedia.org/r/1167926

Change #1167926 merged by Cwhite:

[operations/puppet@production] logstash: fix gitlab event field type conflict

https://gerrit.wikimedia.org/r/1167926

Change #1164526 merged by Cwhite:

[operations/puppet@production] logstash: use filter_on_templates_v2

https://gerrit.wikimedia.org/r/1164526

Change #1167942 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove filter_on_templates v1

https://gerrit.wikimedia.org/r/1167942

Change #1167943 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: rename filter-on-templates.rb

https://gerrit.wikimedia.org/r/1167943

Change #1168201 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: convert numerics - remove field removal and tracking

https://gerrit.wikimedia.org/r/1168201

Change #1168201 merged by Cwhite:

[operations/puppet@production] logstash: convert numerics - remove field removal and tracking

https://gerrit.wikimedia.org/r/1168201

Change #1166008 abandoned by Cwhite:

[operations/software/ecs@master] add docs for string_to_numeric_conversion_failure

Reason:

no longer needed - went another direction

https://gerrit.wikimedia.org/r/1166008

Change #1167942 merged by Cwhite:

[operations/puppet@production] logstash: remove filter_on_templates v1

https://gerrit.wikimedia.org/r/1167942

Change #1167943 merged by Cwhite:

[operations/puppet@production] logstash: rename filter-on-templates.rb

https://gerrit.wikimedia.org/r/1167943