Change Details

It would be useful for us to have a centralised copy of the access logs for dumps.wikimedia.org in order to facilitate analysis. We can create our Nginx log entries in [[https://wikitech.wikimedia.org/wiki/Logstash/Common_Logging_Schema|Elastic Common Schema (ECS)]] format, for maximum compatibility with our Logstash pipeline. There are some examples [[https://www.dash0.com/guides/nginx-logs#configuring-nginx-access-logs-from-plain-text-to-json|here]] of using a custom `log_format` directive to do something similar. We can then use [[https://wikitech.wikimedia.org/wiki/Logstash/Interface#Rsyslog|Rsyslog]] to capture these structured logs and send them to the `kafka-logging` cluster, from where they will be ingested into [[https://wikitech.wikimedia.org/wiki/Logstash|Logstash]]. We can also configure the [[https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/aux-k8s-services/kafka-mirrormaker/values-logging-eqiad-to-jumbo-eqiad.yaml|logging-eqiad-to-jumbo-eqiad]] MirrorMaker to include these logs in the list of those that are mirrored to the `kafka-jumbo` cluster. When they are on kafka-jumbo, we will be able to build a pipeline that pulls these logs and creates hive tables from them. Maybe even index them into Druid, as we do for the `webrequest_sampled_live` tables. == Done is [x] nginx produces Event Platform + ECS compatible events to Kafka [x] logstash consumes these events [x] data lake consumes these events into Hive tables [] 'development' schema and stream naming removed before task is resolved.

It would be useful for us to have a centralised copy of the access logs for dumps.wikimedia.org in order to facilitate analysis. We can create our Nginx log entries in [[https://wikitech.wikimedia.org/wiki/Logstash/Common_Logging_Schema|Elastic Common Schema (ECS)]] format, for maximum compatibility with our Logstash pipeline. There are some examples [[https://www.dash0.com/guides/nginx-logs#configuring-nginx-access-logs-from-plain-text-to-json|here]] of using a custom `log_format` directive to do something similar. We can then use [[https://wikitech.wikimedia.org/wiki/Logstash/Interface#Rsyslog|Rsyslog]] to capture these structured logs and send them to the `kafka-logging` cluster, from where they will be ingested into [[https://wikitech.wikimedia.org/wiki/Logstash|Logstash]]. We can also configure the [[https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/aux-k8s-services/kafka-mirrormaker/values-logging-eqiad-to-jumbo-eqiad.yaml|logging-eqiad-to-jumbo-eqiad]] MirrorMaker to include these logs in the list of those that are mirrored to the `kafka-jumbo` cluster. When they are on kafka-jumbo, we will be able to build a pipeline that pulls these logs and creates hive tables from them. Maybe even index them into Druid, as we do for the `webrequest_sampled_live` tables. == Done is [x] nginx produces Event Platform + ECS compatible events to Kafka [x] logstash consumes these events [x] data lake consumes these events into Hive tables [] canary events are generated for the stream [] 'development' schema and stream naming removed before task is resolved.

It would be useful for us to have a centralised copy of the access logs for dumps.wikimedia.org in order to facilitate analysis. We can create our Nginx log entries in [[https://wikitech.wikimedia.org/wiki/Logstash/Common_Logging_Schema|Elastic Common Schema (ECS)]] format, for maximum compatibility with our Logstash pipeline. There are some examples [[https://www.dash0.com/guides/nginx-logs#configuring-nginx-access-logs-from-plain-text-to-json|here]] of using a custom `log_format` directive to do something similar. We can then use [[https://wikitech.wikimedia.org/wiki/Logstash/Interface#Rsyslog|Rsyslog]] to capture these structured logs and send them to the `kafka-logging` cluster, from where they will be ingested into [[https://wikitech.wikimedia.org/wiki/Logstash|Logstash]]. We can also configure the [[https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/aux-k8s-services/kafka-mirrormaker/values-logging-eqiad-to-jumbo-eqiad.yaml|logging-eqiad-to-jumbo-eqiad]] MirrorMaker to include these logs in the list of those that are mirrored to the `kafka-jumbo` cluster. When they are on kafka-jumbo, we will be able to build a pipeline that pulls these logs and creates hive tables from them. Maybe even index them into Druid, as we do for the `webrequest_sampled_live` tables. == Done is [x] nginx produces Event Platform + ECS compatible events to Kafka [x] logstash consumes these events [x] data lake consumes these events into Hive tables [] canary events are generated for the stream [] 'development' schema and stream naming removed before task is resolved.