Page MenuHomePhabricator

Consider using Elastic Common Schema (ECS) in MediaWiki PSR-3 logs
Open, Needs TriagePublic

Description

ECS is a standardized dictionary of log context keys for Elasticsearch. (E.g. the IP address is always called client.ip; the user agent is always called user_agent.original.) Using standard key names would solve several problems:

  • Within the same Elasticsearch index, a given key name needs to consistently have the same type (string, int etc). If two different log events use the same key with different types, that will result in indexing errors or the field becoming unsearchable. (E.g. this is happening to the user field of many MediaWiki logs now.) Using ECS field names would avoid this for the most common fields.
  • Opensearch dashboards are built from visualisation blocks; visualisations need hard-coded field names. So inconsistency in field names makes visualizations less reusable and Logstash dashboard maintenance more cumbersome.
  • Consistent field names make it easier to create dashboards that display events across multiple sources.
  • Refreshing field mappings is needed less often.

On the other hand, migrating to ECS would require updating tons of dashboards, and there is no easy way to automatically identify dashboards that will be affected.

Event Timeline

Tgr renamed this task from Use Elastic Common Schema (ECS) in MediaWiki PSR-3 logs to Consider using Elastic Common Schema (ECS) in MediaWiki PSR-3 logs.Jun 24 2025, 6:02 PM

I'm pretty sure there was a big discussion about this some years ago, but I couldn't find anything, so filing a new task to discuss it, after it came up again recently in T395204: MediaWiki should log request information (IP, user agent, referrer, HTTP method, etc) in a more uniform and predictable way.

FWIW the output on the Logstash side won't actually be ECS because the logger transforms the keys ( . to _). But that can be fixed in the future, and in any case the problems described above would still be solved.

FWIW the output on the Logstash side won't actually be ECS because the logger transforms the keys ( . to _). But that can be fixed in the future, and in any case the problems described above would still be solved.

This is only an issue if MediaWiki wants to rely on the dot expansion feature of the logging pipeline. This shouldn't be a problem if the generated ECS logs are pre-expanded/nested.

This is only an issue if MediaWiki wants to rely on the dot expansion feature of the logging pipeline. This shouldn't be a problem if the generated ECS logs are pre-expanded/nested.

As in, pass an array to the logger in PHP? (E.g. $logger->info( '...', [ 'user' => [ 'name' => '...' ] ] )?)

I think at least for the migration period that would be worse than the status quo because it would be very easy to get into type conflicts with anything that sends a string for user. Also arrays are somewhat unwieldy (harder to merge contexts for example).

Maybe something like

  1. use ECS names with dot expansion and the current config (so e.g. log user.name which will show up in Logstash as user_name)
  2. ensure nothing uses user in a non-ECS-compatible way
  3. change the Monolog Logstash plugin so it sends both user_name and user.name when it encounters user.name
  4. migrate dashboards from user_name to user.name
  5. change the Monolog Logstash plugin so it only sends user.name when it encounters user.name

would sense?
Granted, that's one more migration step.

The logging pipeline recognizes a log as ECS-formatted when the ecs.version field is set, as required by the docs.

Is it possible to have the logger preserve the dots if ecs.version is a string?