Currently, MediaWiki logs to fluorine (via udp2log), Logstash and Kafka. None of these are currently encrypted in any way.
As a) we would like to log to loghosts across the datacenter boundary and b) logging sometimes includes sensitive information including PII, we should explore ways to encrypt these logging flows.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | herron | T227080 Deprecate all non-Kafka logstash inputs | |||
| Open | None | T205856 Retire udp2log: onboard its producers and consumers to the logging pipeline | |||
| Resolved | LSobanski | T111653 Encrypt all the things | |||
| Open | None | T126989 MediaWiki logging & encryption |
Logging from MediaWiki to Logstash is done via syslog formatted UDP datagrams. Logging from MediaWiki to Fluorine is a UDP datagram flow using the custom udp2log packet encoding. We also use various UDP logging protocols (primarily GELF) to ship debug log events from nodejs and Java applications to Logstash. Adding DTLS support to both client and server for any of these will require non-trivial changes.
It may be easier to introduce a local proxy application that MediaWiki (and other services) can log to using cheap UDP communications and let that service take the brunt of encrypting communications with other endpoints. The lumberjack protocol is one TLS secured method of communicating with Logstash. Rsyslog also has TLS support which could in theory be used to move udp2log datagrams from server to server.
Another idea (Ori's) is to use Kafka for log shipping. Kafka 0.9 (deployed at Wikimedia as we speak) has encryption/authentication/authorization features, we already use it for all of our access log shipping and I believe the Discovery team is already using it from MediaWiki to push logs.
CirrusSearch and the Action API both ship high volume log events to the Hadoop cluster using kafka and strongly typed binary Avro encoded messages. Logstash also consumes a different Kafka topic to publish EventLogging errors in kibana.
@EBernhardson rigged up most of the Monolog->Kafka pipeline.
There is at least one logstash consumer that uses Kafka: https://logstash.wikimedia.org/app/kibana#/dashboard/elasticsearch/eventlogging-errors
All the logs in this dashboard come from a Kafka topic.
Uh, apparently that dashboard doesn't work anymore...(been a long time since I looked at it). Sorry! But it has been done before! :)
Status update: mw logs that were going to logstash in plaintext now are being sent via localhost -> rsyslog -> kafka -> logstash and the network paths are encrypted.
udp2log for mediawiki is still present for debug / high volume logs, however it'll be deprecated in T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline, with end state being that all mw logs are encrypted during transport on the network.
See also https://phabricator.wikimedia.org/T205856#4957430 for the plan (input/feedback welcome!)
Change 494254 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/mediawiki-config@master] WIP: mirror udp2log data into the logging pipeline
Change 492390 had a related patch set uploaded (by Filippo Giunchedi; owner: Herron):
[operations/puppet@production] rsyslog: change udp_localhost_compat to define, add mwlog_compat
Change 498106 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[mediawiki/core@master] [WIP] monolog: add MwlogHandler
Change 498386 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] kafkatee instance for udp2log compat
Change 498387 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] add kafkatee to mwlog
Change 494254 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/mediawiki-config@master] [WIP] mirror udp2log data into the logging pipeline
Change 499734 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logrotate: add old_dir parameter
Change 499734 merged by Filippo Giunchedi:
[operations/puppet@production] logrotate: add old_dir parameter
Change 498386 merged by Filippo Giunchedi:
[operations/puppet@production] profile: kafkatee instance for udp2log compat
Change 498106 merged by Filippo Giunchedi:
[mediawiki/core@master] monolog: add MwlogHandler
I believe with T183303: Decomission old analytics kafka cluster now resolved all mw logs should be encrypted except for udp2log!
Hm, I don't think T183303 affected any encryption status of logs. The Avro logs we migrated to event gate just do an HTTP post to EventGate, and EventGate produces to Kafka unencrypted.
Ah, thanks for the clarification, totally I missed the fact that it'd be HTTP + Kafka in plaintext
Change 494254 abandoned by Filippo Giunchedi:
[operations/mediawiki-config@master] [WIP] mirror udp2log data into the logging pipeline
Reason:
No plans to work on this any time soon
Change #492390 abandoned by Herron:
[operations/puppet@production] rsyslog: change udp_localhost_compat to define, add mwlog_compat
Reason:
spring cleaning -- stale patch