Page MenuHomePhabricator

Standardize the logging format
Open, MediumPublic

Description

A common logging format is desirable for a few reasons (list may not be complete):

  1. to apply an index schema and enable proper type coercion when logs are indexed in Elasticsearch (related incident)
  2. to simplify the log messages and make them easier to reason about and query for
  3. to speed up searches
  4. to simplify de-duplication efforts
  5. to simplify detection and removal of useless logs
  6. to make the upgrade to newer versions of Elasticsearch possible (Single-type mapping per index enforced -- 6.x | 7.x)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+66 -48
integration/configmaster+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+21 -0
operations/puppetproduction+46 -1
operations/puppetproduction+119 -0
operations/puppetproduction+3 -1
operations/puppetproduction+107 -0
operations/puppetproduction+39 -90
operations/puppetproduction+211 -0
operations/puppetproduction+3 -1
operations/puppetproduction+115 -79
operations/puppetproduction+8 -9
operations/puppetproduction+11 -8
operations/puppetproduction+79 -0
operations/puppetproduction+75 -16
operations/puppetproduction+78 -0
operations/puppetproduction+39 -1
operations/puppetproduction+23 -16
operations/puppetproduction+23 -0
operations/puppetproduction+14 -0
operations/puppetproduction+113 -0
operations/puppetproduction+3 K -2
operations/puppetproduction+76 -0
operations/puppetproduction+25 -20
operations/puppetproduction+3 K -1
integration/docrootmaster+5 -0
integration/docrootmaster+5 -0
integration/configmaster+42 -0
integration/configmaster+43 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Thanks @colewhite for starting this! I'm cc'ing @Eevans as I know he's interested in a standardized logging schema too and we've chatted about it in the past as well.

jijiki triaged this task as Medium priority.Oct 14 2019, 2:36 PM

I'm trying to centralise the conversation around field mapping limits and efforts to mitigate/support how we use Logstash in production. Should this be merged into T180051, or perhaps reduced to something that can be a sub task? Or is it orthogonal?

@Krinkle T180051 IMHO implies a different solution. That task, as well as speeding up Kibana, would be accomplished with the work intended here. The last comment from @Eevans lines up with the intent of this task.

This task is complete when we have meaningful, well-documented, and communicated schema definition(s) and corresponding index template(s) installed in ElasticSearch.

Change 604155 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add ecs 1.5.0 template

https://gerrit.wikimedia.org/r/604155

Change 637569 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[integration/config@master] Add ECS build and automated documentation publishing.

https://gerrit.wikimedia.org/r/637569

Change 637569 merged by jenkins-bot:
[integration/config@master] Add ECS build and automated documentation publishing.

https://gerrit.wikimedia.org/r/637569

Mentioned in SAL (#wikimedia-releng) [2020-10-30T17:48:35Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.1-1 # T234565

Change 637743 had a related patch set uploaded (by Hashar; owner: Cwhite):
[integration/config@master] Add jobs for operations/software/ecs

https://gerrit.wikimedia.org/r/637743

Mentioned in SAL (#wikimedia-releng) [2020-10-30T18:16:11Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.2-1 # T234565

Change 637743 merged by jenkins-bot:
[integration/config@master] Add jobs for operations/software/ecs

https://gerrit.wikimedia.org/r/637743

Change 637749 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] Add entry for operations/software/ecs

https://gerrit.wikimedia.org/r/637749

Change 637750 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[integration/docroot@master] add ECS to doc.wm.o index

https://gerrit.wikimedia.org/r/637750

Change 637749 abandoned by Hashar:
[integration/docroot@master] Add entry for operations/software/ecs

Reason:
https://gerrit.wikimedia.org/r/c/integration/docroot/ /637750/1/org/wikimedia/doc/opensource.yaml

https://gerrit.wikimedia.org/r/637749

Change 637750 merged by jenkins-bot:
[integration/docroot@master] add ECS to doc.wm.o index

https://gerrit.wikimedia.org/r/637750

The spam above is @colewhite and I setting up CI to automatically generate https://doc.wikimedia.org/ecs/ . From now on, whenever a patch is merged for the master branch, CI will refresh the documentation. An entry has been added on the index of https://doc.wikimedia.org/

Change 645200 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: make a logstash templates directory and relocate existing templates

https://gerrit.wikimedia.org/r/645200

Change 645209 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add logstash ecs 1.7.0-1 template

https://gerrit.wikimedia.org/r/645209

Change 604155 abandoned by Cwhite:
[operations/puppet@production] profile: add ecs 1.6.0 template

Reason:
superseded

https://gerrit.wikimedia.org/r/604155

Change 645459 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add dot_expander filter script

https://gerrit.wikimedia.org/r/645459

Change 645200 merged by Cwhite:
[operations/puppet@production] profile: make a logstash templates directory and relocate existing templates

https://gerrit.wikimedia.org/r/645200

Change 645459 merged by Cwhite:
[operations/puppet@production] profile: add dot_expander filter script

https://gerrit.wikimedia.org/r/645459

Change 647028 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add ecs pre and post filters to pipeline

https://gerrit.wikimedia.org/r/647028

Change 647029 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add netdev grok patterns to ecs pipeline

https://gerrit.wikimedia.org/r/647029

Change 647032 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: update netdev rsyslog template to ecs 1.7.0

https://gerrit.wikimedia.org/r/647032

Change 645209 merged by Cwhite:
[operations/puppet@production] profile: add logstash ecs 1.7.0-1 template

https://gerrit.wikimedia.org/r/645209

Change 647265 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP logstash: add ulogd ecs filter + tests

https://gerrit.wikimedia.org/r/647265

Change 649956 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add normalize_level filter script

https://gerrit.wikimedia.org/r/649956

Change 650170 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: deploy filter_scripts directory to logstash 7 collectors

https://gerrit.wikimedia.org/r/650170

Change 649956 merged by Cwhite:
[operations/puppet@production] profile: add normalize_level filter script

https://gerrit.wikimedia.org/r/649956

Change 650170 merged by Cwhite:
[operations/puppet@production] profile: deploy filter_scripts directory to logstash 7 collectors

https://gerrit.wikimedia.org/r/650170

Change 657213 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: drop ECS messages on legacy cluster

https://gerrit.wikimedia.org/r/657213

Change 657370 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: enable curator to accept custom age filters

https://gerrit.wikimedia.org/r/657370

Change 657371 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: ecs indices to use a weekly rotation

https://gerrit.wikimedia.org/r/657371

Change 657213 merged by Cwhite:
[operations/puppet@production] profile: drop ECS messages on legacy cluster

https://gerrit.wikimedia.org/r/657213

Change 657370 merged by Cwhite:
[operations/puppet@production] logstash: enable curator to accept custom age filters

https://gerrit.wikimedia.org/r/657370

Change 657371 merged by Cwhite:
[operations/puppet@production] profile: ecs indices to use a weekly rotation

https://gerrit.wikimedia.org/r/657371

Change 647028 merged by Cwhite:
[operations/puppet@production] profile: add ecs pre and post filters to pipeline

https://gerrit.wikimedia.org/r/647028

Change 659422 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: only set default partition if unset

https://gerrit.wikimedia.org/r/659422

Change 659426 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add scap log duplication and ecs mutations

https://gerrit.wikimedia.org/r/659426

Change 661539 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: onboard icinga alerts to common logging schema

https://gerrit.wikimedia.org/r/661539

Change 661539 merged by Cwhite:
[operations/puppet@production] profile: onboard icinga logging to common logging schema

https://gerrit.wikimedia.org/r/661539

Change 647029 merged by Cwhite:
[operations/puppet@production] profile: update netdev to output ECS-formatted logs

https://gerrit.wikimedia.org/r/647029

Change 647032 merged by Cwhite:
[operations/puppet@production] profile: update netdev rsyslog template to ecs 1.7.0

https://gerrit.wikimedia.org/r/647032

Change 663613 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: remove type field for all ecs-formatted events

https://gerrit.wikimedia.org/r/663613

Change 663613 merged by Cwhite:
[operations/puppet@production] profile: remove type field for all ecs-formatted events

https://gerrit.wikimedia.org/r/663613

Change 663876 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add gerrit log duplication and ecs mutations

https://gerrit.wikimedia.org/r/663876

Change 647265 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add ulogd ecs filter + tests

https://gerrit.wikimedia.org/r/647265

Mentioned in SAL (#wikimedia-operations) [2021-02-17T16:46:21Z] <godog> roll-restart logstash to apply ulogd filter - T234565

Mentioned in SAL (#wikimedia-operations) [2021-02-17T17:36:32Z] <godog> roll-restart logstash7 in codfw/eqiad to apply ulogd filters - T234565

Change 659422 merged by Cwhite:
[operations/puppet@production] profile: only set default partition if unset

https://gerrit.wikimedia.org/r/659422

Change 663876 merged by Cwhite:
[operations/puppet@production] profile: add gerrit log duplication and ecs mutations

https://gerrit.wikimedia.org/r/663876

Change 668109 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: swap gerrit log stream to be ecs-only

https://gerrit.wikimedia.org/r/668109

Change 668109 merged by Cwhite:
[operations/puppet@production] profile: swap gerrit log stream to be ecs-only

https://gerrit.wikimedia.org/r/668109

Change 668231 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] httpd: add wmfecsjson logformat to defaults.conf

https://gerrit.wikimedia.org/r/668231

Change 659426 merged by Cwhite:
[operations/puppet@production] profile: add scap log duplication and ecs mutations

https://gerrit.wikimedia.org/r/659426

Change 670525 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: extract index label from logEvent indexing errors

https://gerrit.wikimedia.org/r/670525

Change 670525 merged by Cwhite:
[operations/puppet@production] logstash: extract index label from logEvent indexing errors

https://gerrit.wikimedia.org/r/670525

Change 668231 merged by Cwhite:
[operations/puppet@production] httpd: enable httpd to emit ECS-compliant logs

https://gerrit.wikimedia.org/r/668231

Change 670950 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] phabricator: use ecs-compatible apache log format

https://gerrit.wikimedia.org/r/670950

Change 670951 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] gerrit: use ecs-compatible apache access log format

https://gerrit.wikimedia.org/r/670951

Change 670986 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: rename logEvent exception into error.message

https://gerrit.wikimedia.org/r/670986

Change 670991 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] logstash: grok field name out of error.message

https://gerrit.wikimedia.org/r/670991

Change 670986 merged by Cwhite:
[operations/puppet@production] logstash: rename logEvent exception into error.message

https://gerrit.wikimedia.org/r/670986

Change 670991 merged by Cwhite:
[operations/puppet@production] logstash: grok field name out of error.message

https://gerrit.wikimedia.org/r/670991

Change 670951 merged by Dzahn:
[operations/puppet@production] gerrit: use ecs-compatible apache access log format

https://gerrit.wikimedia.org/r/670951

Change 670950 merged by Dzahn:
[operations/puppet@production] phabricator: use ecs-compatible apache access log format

https://gerrit.wikimedia.org/r/670950

Gerrit and Phabricator now have Apache 2 access log ingested. I have created a saved search in Kibana for source.type: apache2, added a couple very basic visualizations for top source.ip and top domain.name and crafted a dashboard:

https://logstash.wikimedia.org/app/dashboards#/view/825c5c80-8aef-11eb-8ab2-63c7f3b019fc

I have linked it on the main dashboard page as Apache (ECS)

Change 674718 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] logstash: replace ECS allow list with filter_on_template

https://gerrit.wikimedia.org/r/674718

Change 674720 had a related patch set uploaded (by Cwhite; author: Cwhite):
[integration/config@master] mount templates directory for logstash-filter-verifier

https://gerrit.wikimedia.org/r/674720

Change 674720 merged by jenkins-bot:
[integration/config@master] mount templates directory for logstash-filter-verifier

https://gerrit.wikimedia.org/r/674720