T234565: Standardize the logstash logging format - ECS is trying to standardize the software logging format with the Elastic Common Schema. If we are able to produce these ECS logs with Event Platform, they would be automatically ingested into the WMF Data Lake.
This will allow people to use SQL to query the logs with Spark SQL (e.g,. spark3-sql, or pyspark, etc.) or Presto, and can be used for dashboarding with Superset.
This would be particularly useful if we are able to successfully migrate the MediaWiki logging format to ECS, as then MediaWiki software logs could be joined with other MediaWiki data in Hive.
Since the logs would be in Kafka with a well defined schema, they would also be consumable and reusable for other purposes, e.g. stream processing, anomaly detection and alerting, or ingestion into different data stores.
In 2021, Data Engineering and Observability teams met to discuss this idea. To accomplish this, we'd need:
- An ECS+Event Platform compatible event JSONSchema in schemas/event/primary
- This means combining a JSONSchema ECS schema with the Event Platform required fields
- CI Tooling to generate the Elastic index template schema from the ECS event schema. (This Elastic template schema could be stored in operations/software/ecs as it is now, or in schemas/event/primary)
- n.b. I am not sure that we need this this now. Logstash is already ingesting the logs into its ecs-11.1 index.
- Logging streams declared in MediaWiki event stream config
- Logging streams mirrored from the Kafka logging clusters to the Kafka jumbo clusters (T304373: Also intake Network Error Logging events into the Analytics Data Lake also requires this).