Page MenuHomePhabricator
Create Task
Maniphest T205849

Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal)
Closed, ResolvedPublic
Actions

Description

See also goals page for TEC6

  • Procure and provision Logging pipeline hardware in multiple datacenters
  • Migrate >=90% of existing Logstash traffic to the logging pipeline
  • Onboard at least 10 new non-sensitive log producers to the logging pipeline
  • Investigate approaches to ingest sensitive log producers
  • [stretch] Deprecate >= 50% of udp2log producers

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT205849 Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal)
 ResolvedherronT205850 Procure and provision Logging pipeline hardware in multiple datacenters
Unknown Object (Task)
 ResolvedherronT211217 Spin up 3 logstash/kibana frontend VMs in codfw
ResolvedfgiunchediT205851 Migrate >=90% of existing Logstash traffic to the logging pipeline
 ResolvedfgiunchediT211124 Move mediawiki to new logging infrastructure
 ResolvedfgiunchediT205852 Onboard at least 10 new non-sensitive log producers to the logging pipeline
 ResolvedherronT7 Get icinga alerts into logstash
 ResolvedherronT141895 Get phabricator error logs into logstash
 ResolvedherronT209860 Ship peopleweb apache2 error logs to ELK
 OpenNoneT143733 Send Jenkins daemon logs to logstash
 ResolvedherronT210455 Ship prometheus logs to ELK
 ResolvedherronT210458 Ship PuppetDB logs to ELK
OpenFeatureNoneT63780 Add swift logs to logstash
 ResolvedhasharT141324 Look into shoving gerrit logs into logstash
 ResolvedherronT210846 Ship Grafana server logs to ELK
 OpenFeatureNoneT63789 Add Zookeeper log files to logstash
 OpenFeatureNoneT63788 Add kafka log files to logstash
 ResolvedfgiunchediT205855 Investigate approaches to ingest sensitive log producers
 OpenNoneT193766 Ship host syslogs to ELK
 ResolvedNoneT193664 Knock down puppet 4 deprecation warnings
 OpenNoneT197173 Ship MX logs to ELK
 ResolvedfgiunchediT205873 Investigate Kafka main cluster usage for logging pipeline
 ResolvedfgiunchediT203169 Logstash hardware expansion
 ResolvedherronT211065 rack/setup/install codfw logstash elasticsearch storage servers
 ResolvedherronT211859 cronspam from elasticsearch-curator on stretch
 ResolvedherronT206454 Setup Kafka cluster, producers and consumers for logging pipeline
 ResolvedherronT209300 Review and make librdkafka-0.11.6 installable from stretch-wikimedia
 ResolvedfgiunchediT206633 Setup rsyslog to be able to produce logs to Kafka

Event Timeline

fgiunchedi created this task.Oct 1 2018, 12:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2018, 12:53 PM
fgiunchedi added a subtask: T203169: Logstash hardware expansion.Oct 1 2018, 1:11 PM
fgiunchedi mentioned this in T198753: Modernize logging, alerting and metrics monitoring infrastructure - Adopt Logstash (2018-19 Q1 Goal).Oct 1 2018, 1:22 PM
fgiunchedi mentioned this in T200960: Logstash packet loss.Oct 2 2018, 9:10 AM
herron triaged this task as Medium priority.Oct 2 2018, 5:27 PM
fgiunchedi added a project: User-fgiunchedi.Oct 3 2018, 1:15 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
herron mentioned this in T205873: Investigate Kafka main cluster usage for logging pipeline.Oct 4 2018, 7:36 PM
herron mentioned this in T203169: Logstash hardware expansion.Oct 4 2018, 8:44 PM
fgiunchedi closed subtask T205873: Investigate Kafka main cluster usage for logging pipeline as Resolved.Oct 8 2018, 10:07 AM
fgiunchedi moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.Oct 15 2018, 2:24 PM
akosiaris mentioned this in T207200: Revisit the logging work done on Q1 2017-2018 for the standard pod setup.Oct 16 2018, 6:27 PM
• mobrovac added projects: Platform Team Legacy (Watching / External), Services (watching).Oct 30 2018, 4:45 PM
gerritbot subscribed.Nov 5 2018, 2:11 PM

Change 471733 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hierdata: test rollout of rsyslog_exporter

https://gerrit.wikimedia.org/r/471733

gerritbot added a project: Patch-For-Review.Nov 5 2018, 2:11 PM
gerritbot added a comment.Nov 5 2018, 2:53 PM

Change 471733 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: test rollout of rsyslog_exporter

https://gerrit.wikimedia.org/r/471733

gerritbot added a comment.Nov 7 2018, 3:23 PM

Change 472165 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: expand rsyslog_exporter rollout

https://gerrit.wikimedia.org/r/472165

gerritbot added a comment.Nov 7 2018, 3:32 PM

Change 472165 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: expand rsyslog_exporter rollout

https://gerrit.wikimedia.org/r/472165

gerritbot added a comment.Nov 8 2018, 9:20 AM

Change 472395 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in ulsfo

https://gerrit.wikimedia.org/r/472395

gerritbot added a comment.Nov 8 2018, 9:20 AM

Change 472396 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in codfw

https://gerrit.wikimedia.org/r/472396

gerritbot added a comment.Nov 8 2018, 9:49 AM

Change 472395 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in ulsfo

https://gerrit.wikimedia.org/r/472395

gerritbot added a comment.Nov 8 2018, 10:29 AM

Change 472403 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in eqsin

https://gerrit.wikimedia.org/r/472403

gerritbot added a comment.Nov 8 2018, 10:29 AM

Change 472404 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout rsyslog_exporter in esams

https://gerrit.wikimedia.org/r/472404

gerritbot added a comment.Nov 8 2018, 10:41 AM

Change 472408 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add rsyslog_exporter jobs

https://gerrit.wikimedia.org/r/472408

gerritbot added a comment.Nov 8 2018, 10:52 AM

Change 472408 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add rsyslog_exporter jobs

https://gerrit.wikimedia.org/r/472408

gerritbot added a comment.Nov 8 2018, 1:05 PM

Change 472404 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in esams

https://gerrit.wikimedia.org/r/472404

gerritbot added a comment.Nov 8 2018, 1:49 PM

Change 472403 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in eqsin

https://gerrit.wikimedia.org/r/472403

gerritbot added a comment.Nov 12 2018, 8:23 AM

Change 472396 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout rsyslog_exporter in codfw

https://gerrit.wikimedia.org/r/472396

gerritbot added a comment.Nov 12 2018, 9:16 AM

Change 472948 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: move enable_rsyslog_exporter into hieradata/profile

https://gerrit.wikimedia.org/r/472948

gerritbot added a comment.Nov 12 2018, 9:32 AM

Change 472948 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: enable rsyslog_exporter in ulsfo

https://gerrit.wikimedia.org/r/472948

herron added a project: User-herron.Nov 12 2018, 5:01 PM
herron moved this task from Backlog to Working on on the User-herron board.Nov 19 2018, 3:09 PM
herron closed subtask T206454: Setup Kafka cluster, producers and consumers for logging pipeline as Resolved.Nov 19 2018, 3:19 PM
fgiunchedi closed subtask T206633: Setup rsyslog to be able to produce logs to Kafka as Resolved.Dec 4 2018, 1:06 PM
fgiunchedi mentioned this in T211124: Move mediawiki to new logging infrastructure.Dec 4 2018, 4:12 PM
• chasemp subscribed.Dec 13 2018, 3:53 PM
herron closed subtask T205850: Procure and provision Logging pipeline hardware in multiple datacenters as Resolved.Dec 17 2018, 10:11 PM
gerritbot added a comment.Dec 19 2018, 4:16 PM

Change 480787 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash::collector: pull logs from both kafka-logging clusters

https://gerrit.wikimedia.org/r/480787

gerritbot added a comment.Dec 19 2018, 4:43 PM

Change 480791 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash::collector add input identifier tags

https://gerrit.wikimedia.org/r/480791

gerritbot added a comment.Dec 19 2018, 4:58 PM

Change 480793 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog::kafka_shipper: set rsyslog MaxMessageSize to 64k

https://gerrit.wikimedia.org/r/480793

fgiunchedi closed subtask T205851: Migrate >=90% of existing Logstash traffic to the logging pipeline as Resolved.Jan 16 2019, 10:18 AM
fgiunchedi closed this task as Resolved.Jan 16 2019, 10:24 AM
fgiunchedi claimed this task.
fgiunchedi updated the task description. (Show Details)

Resolving, though some subtasks will be part of T213157: Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6)

fgiunchedi removed a subtask: T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline.Jan 16 2019, 11:16 AM
fgiunchedi closed subtask T205855: Investigate approaches to ingest sensitive log producers as Resolved.
fgiunchedi closed subtask T205852: Onboard at least 10 new non-sensitive log producers to the logging pipeline as Resolved.Jan 16 2019, 11:18 AM
fgiunchedi closed subtask T203169: Logstash hardware expansion as Resolved.Jan 16 2019, 11:33 AM
gerritbot added a comment.Feb 4 2019, 9:35 PM

Change 480791 merged by Herron:
[operations/puppet@production] logstash::collector add input identifier tags

https://gerrit.wikimedia.org/r/480791

gerritbot added a comment.Feb 5 2019, 9:20 PM

Change 480793 merged by Herron:
[operations/puppet@production] rsyslog::kafka_shipper: raise rsyslog MaxMessageSize from 8k to 64k

https://gerrit.wikimedia.org/r/480793

gerritbot added a comment.Feb 12 2019, 3:41 PM

Change 480787 merged by Herron:
[operations/puppet@production] logstash::collector: pull logs from both kafka-logging clusters

https://gerrit.wikimedia.org/r/480787

fgiunchedi added a project: observability.Aug 19 2019, 2:31 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 3:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL