Several Wikimedia deployed services (parsoid, OCG, restbase, ...) implement logging by sending GELF packets to the ELK cluster for indexing. A common request heard in #mediawiki-parsoid and other irc channels is for some sort of export of data from the backing Elasticsearch cluster to help track down some complex operational problem. If we had a udp2log compatible output plugin for Logstash we could tag events from these services, route them to the output and end up with text logs on fluorinemwlog1001.eqiad.wmnet.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Nowadays we're deprecating udp2log, though the issue of sending logs from logstash out elsewhere still stands. cc Parsing-Team--ARCHIVED for their opinion on what sort of log export they would like to see
Comment Actions
udp2log has been removed from its former role in the varnish/pageviews pipeline, but it is still very much actively used by MediaWiki in production to receive log events on mwlog1001.eqiad.wmnet. If the SRE team has plans to fully remove udp2log in production investigations will need to be done on a replacement for the logs which are shipped to mwlog1001.eqiad.wmnet and known to be to large/frequent for the existing ELK cluster to ingest.
Comment Actions
Agreed, with "deprecation" I meant more "no new use cases" for insecure transports like udp2log.
Comment Actions
There's now an explicit goal to retire udp2log from production: T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline.
Comment Actions
Boldly resolving this task, with the logging pipeline in production we can either tap into the kafka log stream pre-logstash or inject messages back into kafka post-logstash after processing