Page MenuHomePhabricator

MediaWiki logging & encryption
Open, HighPublic

Description

Currently, MediaWiki logs to fluorine (via udp2log), Logstash and Kafka. None of these are currently encrypted in any way.

As a) we would like to log to loghosts across the datacenter boundary and b) logging sometimes includes sensitive information including PII, we should explore ways to encrypt these logging flows.

Details

Related Gerrit Patches:
mediawiki/core : mastermonolog: add MwlogHandler
operations/puppet : productionprofile: kafkatee instance for udp2log compat
operations/puppet : productionlogrotate: add old_dir parameter
operations/mediawiki-config : master[WIP] mirror udp2log data into the logging pipeline
operations/puppet : production[WIP] add kafkatee to mwlog
operations/puppet : productionrsyslog: change udp_localhost_compat to define, add mwlog_compat

Event Timeline

faidon created this task.Feb 15 2016, 4:44 PM
faidon raised the priority of this task from to High.
faidon updated the task description. (Show Details)
faidon set Security to None.
faidon added subscribers: Volker_E, Legoktm, JanZerebecki and 8 others.

Logging from MediaWiki to Logstash is done via syslog formatted UDP datagrams. Logging from MediaWiki to Fluorine is a UDP datagram flow using the custom udp2log packet encoding. We also use various UDP logging protocols (primarily GELF) to ship debug log events from nodejs and Java applications to Logstash. Adding DTLS support to both client and server for any of these will require non-trivial changes.

It may be easier to introduce a local proxy application that MediaWiki (and other services) can log to using cheap UDP communications and let that service take the brunt of encrypting communications with other endpoints. The lumberjack protocol is one TLS secured method of communicating with Logstash. Rsyslog also has TLS support which could in theory be used to move udp2log datagrams from server to server.

Another idea (Ori's) is to use Kafka for log shipping. Kafka 0.9 (deployed at Wikimedia as we speak) has encryption/authentication/authorization features, we already use it for all of our access log shipping and I believe the Discovery team is already using it from MediaWiki to push logs.

Another idea (Ori's) is to use Kafka for log shipping. Kafka 0.9 (deployed at Wikimedia as we speak) has encryption/authentication/authorization features, we already use it for all of our access log shipping and I believe the Discovery team is already using it from MediaWiki to push logs.

CirrusSearch and the Action API both ship high volume log events to the Hadoop cluster using kafka and strongly typed binary Avro encoded messages. Logstash also consumes a different Kafka topic to publish EventLogging errors in kibana.

@EBernhardson rigged up most of the Monolog->Kafka pipeline.

I don't think we have encryption for kafka yet ? I'd love to be wrong though cc @Ottomata @elukey
Once we have kafka encryption we could use that for shipping mw logs too to logstash

Timeline for this just discussed and noted in T152015

There is at least one logstash consumer that uses Kafka: https://logstash.wikimedia.org/app/kibana#/dashboard/elasticsearch/eventlogging-errors

All the logs in this dashboard come from a Kafka topic.

Uh, apparently that dashboard doesn't work anymore...(been a long time since I looked at it). Sorry! But it has been done before! :)

Status update: mw logs that were going to logstash in plaintext now are being sent via localhost -> rsyslog -> kafka -> logstash and the network paths are encrypted.

udp2log for mediawiki is still present for debug / high volume logs, however it'll be deprecated in T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline, with end state being that all mw logs are encrypted during transport on the network.

CDanis awarded a token.Feb 7 2019, 1:41 PM
CDanis rescinded a token.
CDanis awarded a token.

Status update: mw logs that were going to logstash in plaintext now are being sent via localhost -> rsyslog -> kafka -> logstash and the network paths are encrypted.
udp2log for mediawiki is still present for debug / high volume logs, however it'll be deprecated in T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline, with end state being that all mw logs are encrypted during transport on the network.

See also https://phabricator.wikimedia.org/T205856#4957430 for the plan (input/feedback welcome!)

Change 494254 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/mediawiki-config@master] WIP: mirror udp2log data into the logging pipeline

https://gerrit.wikimedia.org/r/494254

Change 492390 had a related patch set uploaded (by Filippo Giunchedi; owner: Herron):
[operations/puppet@production] rsyslog: change udp_localhost_compat to define, add mwlog_compat

https://gerrit.wikimedia.org/r/492390

Change 498106 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[mediawiki/core@master] [WIP] monolog: add MwlogHandler

https://gerrit.wikimedia.org/r/498106

Change 498386 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] kafkatee instance for udp2log compat

https://gerrit.wikimedia.org/r/498386

Change 498387 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] add kafkatee to mwlog

https://gerrit.wikimedia.org/r/498387

Change 498387 abandoned by Filippo Giunchedi:
[WIP] add kafkatee to mwlog

https://gerrit.wikimedia.org/r/498387

Change 494254 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/mediawiki-config@master] [WIP] mirror udp2log data into the logging pipeline

https://gerrit.wikimedia.org/r/494254

Change 499734 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logrotate: add old_dir parameter

https://gerrit.wikimedia.org/r/499734

Change 499734 merged by Filippo Giunchedi:
[operations/puppet@production] logrotate: add old_dir parameter

https://gerrit.wikimedia.org/r/499734

Change 498386 merged by Filippo Giunchedi:
[operations/puppet@production] profile: kafkatee instance for udp2log compat

https://gerrit.wikimedia.org/r/498386

Change 498106 merged by Filippo Giunchedi:
[mediawiki/core@master] monolog: add MwlogHandler

https://gerrit.wikimedia.org/r/498106

I believe with T183303: Decomission old analytics kafka cluster now resolved all mw logs should be encrypted except for udp2log!

Hm, I don't think T183303 affected any encryption status of logs. The Avro logs we migrated to event gate just do an HTTP post to EventGate, and EventGate produces to Kafka unencrypted.

Hm, I don't think T183303 affected any encryption status of logs. The Avro logs we migrated to event gate just do an HTTP post to EventGate, and EventGate produces to Kafka unencrypted.

Ah, thanks for the clarification, totally I missed the fact that it'd be HTTP + Kafka in plaintext