Page MenuHomePhabricator

Make a udp2log output plugin for Logstash
Closed, DeclinedPublic

Description

Several Wikimedia deployed services (parsoid, OCG, restbase, ...) implement logging by sending GELF packets to the ELK cluster for indexing. A common request heard in #mediawiki-parsoid and other irc channels is for some sort of export of data from the backing Elasticsearch cluster to help track down some complex operational problem. If we had a udp2log compatible output plugin for Logstash we could tag events from these services, route them to the output and end up with text logs on fluorinemwlog1001.eqiad.wmnet.

Event Timeline

bd808 raised the priority of this task from to Medium.
bd808 updated the task description. (Show Details)
bd808 added a project: MediaWiki-Debug-Logger.
bd808 added a subscriber: bd808.

Nowadays we're deprecating udp2log, though the issue of sending logs from logstash out elsewhere still stands. cc Parsing-Team--ARCHIVED for their opinion on what sort of log export they would like to see

udp2log has been removed from its former role in the varnish/pageviews pipeline, but it is still very much actively used by MediaWiki in production to receive log events on mwlog1001.eqiad.wmnet. If the SRE team has plans to fully remove udp2log in production investigations will need to be done on a replacement for the logs which are shipped to mwlog1001.eqiad.wmnet and known to be to large/frequent for the existing ELK cluster to ingest.

Agreed, with "deprecation" I meant more "no new use cases" for insecure transports like udp2log.

udp2log has been removed from its former role in the varnish/pageviews pipeline, but it is still very much actively used by MediaWiki in production to receive log events on mwlog1001.eqiad.wmnet. If the SRE team has plans to fully remove udp2log in production investigations will need to be done on a replacement for the logs which are shipped to mwlog1001.eqiad.wmnet and known to be to large/frequent for the existing ELK cluster to ingest.

There's now an explicit goal to retire udp2log from production: T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline.

Boldly resolving this task, with the logging pipeline in production we can either tap into the kafka log stream pre-logstash or inject messages back into kafka post-logstash after processing