Page MenuHomePhabricator

Standardize the logging format
Open, MediumPublic

Description

A common logging format is desirable for a few reasons (list may not be complete):

  1. to apply an index schema and enable proper type coercion when logs are indexed in Elasticsearch (related incident)
  2. to simplify the log messages and make them easier to reason about and query for
  3. to speed up searches
  4. to simplify de-duplication efforts
  5. to simplify detection and removal of useless logs
  6. to make the upgrade to newer versions of Elasticsearch possible (Single-type mapping per index enforced -- 6.x | 7.x)

Event Timeline

colewhite created this task.Oct 3 2019, 8:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 3 2019, 8:24 PM
colewhite updated the task description. (Show Details)Oct 3 2019, 8:41 PM

Thanks @colewhite for starting this! I'm cc'ing @Eevans as I know he's interested in a standardized logging schema too and we've chatted about it in the past as well.

jijiki triaged this task as Medium priority.Oct 14 2019, 2:36 PM
Krinkle added a subscriber: Krinkle.Nov 7 2019, 5:54 PM

I'm trying to centralise the conversation around field mapping limits and efforts to mitigate/support how we use Logstash in production. Should this be merged into T180051, or perhaps reduced to something that can be a sub task? Or is it orthogonal?

@Krinkle T180051 IMHO implies a different solution. That task, as well as speeding up Kibana, would be accomplished with the work intended here. The last comment from @Eevans lines up with the intent of this task.

This task is complete when we have meaningful, well-documented, and communicated schema definition(s) and corresponding index template(s) installed in ElasticSearch.

fgiunchedi moved this task from Inbox to In progress on the observability board.Mar 16 2020, 2:22 PM

Change 604155 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add ecs 1.5.0 template

https://gerrit.wikimedia.org/r/604155

Change 637569 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[integration/config@master] Add ECS build and automated documentation publishing.

https://gerrit.wikimedia.org/r/637569

Change 637569 merged by jenkins-bot:
[integration/config@master] Add ECS build and automated documentation publishing.

https://gerrit.wikimedia.org/r/637569

Mentioned in SAL (#wikimedia-releng) [2020-10-30T17:48:35Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.1-1 # T234565

Change 637743 had a related patch set uploaded (by Hashar; owner: Cwhite):
[integration/config@master] Add jobs for operations/software/ecs

https://gerrit.wikimedia.org/r/637743

Mentioned in SAL (#wikimedia-releng) [2020-10-30T18:16:11Z] <hashar> Successfully tagged docker-registry.discovery.wmnet/releng/ecs:0.0.2-1 # T234565

Change 637743 merged by jenkins-bot:
[integration/config@master] Add jobs for operations/software/ecs

https://gerrit.wikimedia.org/r/637743

Change 637749 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/docroot@master] Add entry for operations/software/ecs

https://gerrit.wikimedia.org/r/637749

Change 637750 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[integration/docroot@master] add ECS to doc.wm.o index

https://gerrit.wikimedia.org/r/637750

Change 637749 abandoned by Hashar:
[integration/docroot@master] Add entry for operations/software/ecs

Reason:
https://gerrit.wikimedia.org/r/c/integration/docroot/ /637750/1/org/wikimedia/doc/opensource.yaml

https://gerrit.wikimedia.org/r/637749

Change 637750 merged by jenkins-bot:
[integration/docroot@master] add ECS to doc.wm.o index

https://gerrit.wikimedia.org/r/637750

hashar added a subscriber: hashar.Fri, Oct 30, 6:30 PM

The spam above is @colewhite and I setting up CI to automatically generate https://doc.wikimedia.org/ecs/ . From now on, whenever a patch is merged for the master branch, CI will refresh the documentation. An entry has been added on the index of https://doc.wikimedia.org/