Page MenuHomePhabricator

Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal)
Closed, ResolvedPublic

Description

See also goals page for TEC6

  • Procure and provision Logging pipeline hardware in multiple datacenters
  • Migrate >=90% of existing Logstash traffic to the logging pipeline
  • Onboard at least 10 new non-sensitive log producers to the logging pipeline
  • Investigate approaches to ingest sensitive log producers
  • [stretch] Deprecate >= 50% of udp2log producers

Related Objects

StatusAssignedTask
Resolvedfgiunchedi
Resolvedherron
Resolvedherron
Resolvedfgiunchedi
Resolvedfgiunchedi
OpenPchelolo
OpenMvolz
OpenNone
OpenOttomata
StalledNone
OpenNone
OpenNone
Resolvedbmansurov
ResolvedArlolra
OpenNone
OpenNone
Resolvedfgiunchedi
Resolvedherron
Resolvedherron
Resolvedherron
OpenNone
Resolvedherron
Resolvedherron
OpenNone
OpenPaladox
Resolvedherron
OpenNone
OpenNone
Resolvedfgiunchedi
OpenNone
OpenNone
OpenNone
Resolvedfgiunchedi
Resolvedfgiunchedi
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedherron
Resolvedfgiunchedi

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 12:53 PM
herron triaged this task as Normal priority.Oct 2 2018, 5:27 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.

Change 471733 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hierdata: test rollout of rsyslog_exporter

https://gerrit.wikimedia.org/r/471733

Change 471733 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: test rollout of rsyslog_exporter

https://gerrit.wikimedia.org/r/471733

Change 472165 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: expand rsyslog_exporter rollout

https://gerrit.wikimedia.org/r/472165

Change 472165 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: expand rsyslog_exporter rollout

https://gerrit.wikimedia.org/r/472165

Change 472395 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in ulsfo

https://gerrit.wikimedia.org/r/472395

Change 472396 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in codfw

https://gerrit.wikimedia.org/r/472396

Change 472395 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in ulsfo

https://gerrit.wikimedia.org/r/472395

Change 472403 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in eqsin

https://gerrit.wikimedia.org/r/472403

Change 472404 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in esams

https://gerrit.wikimedia.org/r/472404

Change 472408 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add rsyslog_exporter jobs

https://gerrit.wikimedia.org/r/472408

Change 472408 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add rsyslog_exporter jobs

https://gerrit.wikimedia.org/r/472408

Change 472404 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in esams

https://gerrit.wikimedia.org/r/472404

Change 472403 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in eqsin

https://gerrit.wikimedia.org/r/472403

Change 472396 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in codfw

https://gerrit.wikimedia.org/r/472396

Change 472948 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: move enable_rsyslog_exporter into hieradata/profile

https://gerrit.wikimedia.org/r/472948

Change 472948 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: enable rsyslog_exporter in ulsfo

https://gerrit.wikimedia.org/r/472948

herron moved this task from Backlog to Working on on the User-herron board.Nov 19 2018, 3:09 PM

Change 480787 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash::collector: pull logs from both kafka-logging clusters

https://gerrit.wikimedia.org/r/480787

Change 480791 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash::collector add input identifier tags

https://gerrit.wikimedia.org/r/480791

Change 480793 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog::kafka_shipper: set rsyslog MaxMessageSize to 64k

https://gerrit.wikimedia.org/r/480793

fgiunchedi updated the task description. (Show Details)Jan 16 2019, 10:24 AM
fgiunchedi closed this task as Resolved.
fgiunchedi claimed this task.

Change 480791 merged by Herron:
[operations/puppet@production] logstash::collector add input identifier tags

https://gerrit.wikimedia.org/r/480791

Change 480793 merged by Herron:
[operations/puppet@production] rsyslog::kafka_shipper: raise rsyslog MaxMessageSize from 8k to 64k

https://gerrit.wikimedia.org/r/480793

Change 480787 merged by Herron:
[operations/puppet@production] logstash::collector: pull logs from both kafka-logging clusters

https://gerrit.wikimedia.org/r/480787