Page MenuHomePhabricator
Create Task
Maniphest T227080

Deprecate all non-Kafka logstash inputs
Closed, ResolvedPublic
Actions

Description

Umbrella task to track status of applications using non-kafka inputs and thus need to be migrated to the new logging infrastructure.

See also "logstash server statistics" Kibana dashboard for a breakdown of kafka vs non-kafka input traffic https://logstash.wikimedia.org/goto/713a67c3f6ae7ff50a2595d92fde9c16

Details

SubjectRepoBranchLines +/-
scap: enable logging to syslogoperations/puppetproduction+2 -0
scap: disable udp loggingoperations/puppetproduction+0 -2
logstash: write kafka inputs to es by defaultoperations/puppetproduction+17 -17
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT224602 Fix logging umbrella task
 ResolvedherronT227080 Deprecate all non-Kafka logstash inputs
 OpenNoneT205856 Retire udp2log: onboard its producers and consumers to the logging pipeline
 ResolvedfgiunchediT86969 Send scap log directly to logstash via syslog input
 OpenNoneT215497 Move iegreview from udp2log to syslog
 ResolvedfgiunchediT215499 Move wikimania-scholarships from udp2log to syslog
 OpenNoneT126989 MediaWiki logging & encryption
 ResolvedfgiunchediT213899 Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs
 Resolvedbd808T217932 Change log routing to ELK cluster to use rsyslog->kafka rather than talking directly to the ELK cluster
 ResolvedEBernhardsonT218833 Migrate mjolnir to stdout/syslog/cee logging output
 ResolvedfgiunchediT227108 Port varnishlog consumers to log to syslog / logging infra
 ResolvedfgiunchediT242585 Move cassandra logging to logging pipeline
 ResolvedfgiunchediT242609 Move thumbor to the logging pipeline
 InvalidNoneT225122 Migrate services using deprecated Gelf logstash input to Kafka enabled logging pipeline
 ResolvedherronT225125 Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline
 Resolved• PcheloloT211125 Move service-runner to new logging infrastructure
 ResolvedMvolzT219919 Move citoid logging to new logging pipeline
 ResolvedakosiarisT219921 Move cxserver logging to new logging pipeline
 DeclinedOttomataT219922 Move eventstreams logging to new logging pipeline
 DeclinedNoneT219923 Move graphoid logging to new logging pipeline
 ResolvedfgiunchediT219924 Move mobileapps logging to new logging pipeline
 ResolvedfgiunchediT219925 Move proton logging to new logging pipeline
 Resolved• bmansurovT219926 Move recommendation-api logging to new logging pipeline
 ResolvedArlolraT219927 Move parsoid logging to new logging pipeline
 ResolvedelukeyT219928 Move AQS logging to new logging pipeline
 ResolvedMilimetricT236757 Upgrade service-runner on AQS to unblock rsyslog logging
 ResolvedMSantosT222377 Move kartotherian/tilerator logging to new logging pipeline
 ResolvedOttomataT225129 Move eventgate logs to new logging infrastructure
 ResolvedfgiunchediT245511 Move netbox uwsgi logs to logging pipeline
 ResolvedfgiunchediT245512 Move service::uwsgi logs to logging pipeline
 DeclinedNoneT245515 Move restrouter to the logging pipeline
 ResolvedakosiarisT245516 Move mathoid to the logging pipeline
 ResolvedfgiunchediT245603 Move termbox to the logging pipeline
 ResolvedfgiunchediT245604 Move wikifeeds to the logging pipeline
 ResolvedakosiarisT239459 service-runner apps running on kubernetes emit logs with log level 50

Event Timeline

fgiunchedi created this task.Jul 2 2019, 12:46 PM
fgiunchedi added a subtask: T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline.Jul 2 2019, 12:49 PM
fgiunchedi removed a parent task: T220103: TEC6: Logging infrastructure (Q4 2018/19 goal).
fgiunchedi added a subtask: T213899: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs.Jul 2 2019, 12:51 PM
fgiunchedi removed a project: Goal.
fgiunchedi added a subtask: T211125: Move service-runner to new logging infrastructure.Jul 2 2019, 12:54 PM
fgiunchedi updated the task description. (Show Details)Jul 2 2019, 1:21 PM
MoritzMuehlenhoff triaged this task as Medium priority.Jul 3 2019, 7:59 AM
fgiunchedi added a project: observability.Aug 19 2019, 2:29 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jan 13 2020, 10:08 AM
fgiunchedi added a subtask: T225122: Migrate services using deprecated Gelf logstash input to Kafka enabled logging pipeline.Jan 13 2020, 2:30 PM
fgiunchedi closed subtask T242609: Move thumbor to the logging pipeline as Resolved.Jan 21 2020, 11:39 AM
fgiunchedi closed subtask T242585: Move cassandra logging to logging pipeline as Resolved.Feb 11 2020, 9:21 AM
fgiunchedi closed subtask T227108: Port varnishlog consumers to log to syslog / logging infra as Resolved.Feb 18 2020, 1:40 PM
fgiunchedi added a subtask: T245511: Move netbox uwsgi logs to logging pipeline.Feb 18 2020, 1:47 PM
fgiunchedi closed subtask T213899: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs as Resolved.Feb 18 2020, 2:05 PM
fgiunchedi mentioned this in T213899: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs.
akosiaris closed subtask T245516: Move mathoid to the logging pipeline as Resolved.Feb 19 2020, 9:52 AM
akosiaris closed subtask T245515: Move restrouter to the logging pipeline as Declined.Feb 19 2020, 10:23 AM
fgiunchedi closed subtask T225122: Migrate services using deprecated Gelf logstash input to Kafka enabled logging pipeline as Invalid.Feb 24 2020, 9:39 AM
gerritbot added a comment.Feb 24 2020, 2:26 PM

Change 574467 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: write kafka inputs to es by default

https://gerrit.wikimedia.org/r/574467

gerritbot added a project: Patch-For-Review.Feb 24 2020, 2:26 PM
gerritbot added a comment.Feb 24 2020, 3:21 PM

Change 574485 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] scap: enable logging to syslog

https://gerrit.wikimedia.org/r/574485

gerritbot added a comment.Feb 25 2020, 1:41 PM

Change 574467 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: write kafka inputs to es by default

https://gerrit.wikimedia.org/r/574467

Stashbot added a comment.Feb 25 2020, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2020-02-25T13:42:05Z] <godog> roll-restart logstash in eqiad/codfw - T227080

fgiunchedi closed subtask T245511: Move netbox uwsgi logs to logging pipeline as Resolved.Feb 25 2020, 2:50 PM
fgiunchedi closed subtask T245512: Move service::uwsgi logs to logging pipeline as Resolved.Feb 25 2020, 4:50 PM
Jdforrester-WMF mentioned this in T246937: deployment-puppetmaster04 disc use exploding .Mar 4 2020, 9:20 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Apr 6 2020, 12:36 PM
akosiaris added a subtask: T239459: service-runner apps running on kubernetes emit logs with log level 50 .Apr 15 2020, 4:56 PM
fgiunchedi moved this task from Doing to Up next on the User-fgiunchedi board.Apr 27 2020, 12:12 PM
fgiunchedi closed subtask T245604: Move wikifeeds to the logging pipeline as Resolved.Sep 21 2020, 9:32 AM
fgiunchedi reopened subtask T245604: Move wikifeeds to the logging pipeline as Open.Sep 21 2020, 9:35 AM
fgiunchedi closed subtask T245604: Move wikifeeds to the logging pipeline as Resolved.Oct 6 2020, 7:40 AM
fgiunchedi added a comment.Oct 6 2020, 7:52 AM

As far as gelf is concerned we're left with:

  • elasticsearch: (T225125) pending ES7 upgrade to get native json logging
  • graphoid: will be undeployed
  • kartotherian: (T222377) blocked by unrelated changes pending deployment
fgiunchedi closed subtask T245603: Move termbox to the logging pipeline as Resolved.Oct 7 2020, 1:58 PM
akosiaris closed subtask T239459: service-runner apps running on kubernetes emit logs with log level 50 as Resolved.Jan 13 2021, 3:49 PM
gerritbot added a comment.Jan 19 2021, 4:16 PM

Change 657136 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] scap: disable udp logging

https://gerrit.wikimedia.org/r/657136

gerritbot added a comment.Jan 21 2021, 7:37 AM

Change 574485 merged by Effie Mouzeli:
[operations/puppet@production] scap: enable logging to syslog

https://gerrit.wikimedia.org/r/574485

gerritbot added a comment.Jan 21 2021, 7:40 AM

Change 657136 merged by Effie Mouzeli:
[operations/puppet@production] scap: disable udp logging

https://gerrit.wikimedia.org/r/657136

Maintenance_bot removed a project: Patch-For-Review.Jan 21 2021, 8:10 AM
gerritbot added a comment.Feb 11 2021, 10:38 PM

Change 663689 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[wikimedia/slimapp@master] use monolog SyslogHandler and add CeeFormatter

https://gerrit.wikimedia.org/r/663689

gerritbot added a project: Patch-For-Review.Feb 11 2021, 10:38 PM
fgiunchedi moved this task from Up next to Backlog on the User-fgiunchedi board.Apr 27 2021, 1:46 PM
fgiunchedi removed a project: User-fgiunchedi.May 6 2021, 12:29 PM
• Pchelolo closed subtask T211125: Move service-runner to new logging infrastructure as Resolved.May 25 2021, 8:50 PM
lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:39 AM
Maintenance_bot added a project: observability.Jul 12 2021, 2:48 AM
lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM
fgiunchedi updated the task description. (Show Details)Jul 20 2021, 9:22 AM
herron mentioned this in T281266: Decommission old ELK5 Logstash cluster.Jul 27 2021, 5:39 PM
lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 1:13 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 1:46 AM
lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 2:36 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 2:46 AM
lmata edited projects, added Observability-Logging; removed SRE Observability.Aug 9 2021, 3:23 AM
Maintenance_bot edited projects, added SRE Observability; removed Observability-Logging.Aug 9 2021, 3:48 AM
• nnikkhoui mentioned this in T290156: Config.dev.yaml and config.prod.yaml need updating to remove references to logstash.Sep 1 2021, 1:04 PM
herron closed this task as Resolved.Dec 7 2021, 8:51 PM
herron claimed this task.
herron subscribed.

Looking at the dashboard linked in the description there have been no logs received via legacy inputs over the past 4 days. I think we're in a good place to resolve this now, and if any follow up is needed we can reopen.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL