The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

When they are on kafka-jumbo, we will be able to build a pipeline that pulls these logs and creates hive tables from them.

Probably out of scope I knoooowww but wouldn't T291645: Produce ECS formatted logstash logs to Event Platform, allowing them to be queried in the WMF Data Lake with SQL be nice?

1. make an ECS compatible event schema
2. declare an event stream in EventStreamConfig
3. produce logs matching schema (there are details here) to stream in kafka jumbo-eqiad
4. Logs are automatically ingested to Hive tables, no bespoke ingestion pipelines needed.

See also T390323: Data Platform, SRE Observability, overlaps, use cases, and potential

In T425087#11879870, @Ottomata wrote:

When they are on kafka-jumbo, we will be able to build a pipeline that pulls these logs and creates hive tables from them.

Probably out of scope I knoooowww but wouldn't T291645: Produce ECS formatted logstash logs to Event Platform, allowing them to be queried in the WMF Data Lake with SQL be nice?

1. make an ECS compatible event schema
2. declare an event stream in EventStreamKafka
3. produce logs matching schema (there are details here) to stream in kafka jumbo-eqiad
4. Logs are automatically ingested to Hive tables, no bespoke ingestion pipelines needed.

See also T390323: Data Platform, SRE Observability, overlaps, use cases, and potential

OK, this sounds good. We already have the required mirrormaker config to mirror any eventstream topics (any topics prefixed with eqiad. or codfw.) from kafka-logging-[eqiad|codfw] to kafka-jumbo-eqiad.

i can have a crack at the ECS event schema, although I will probably need some hand-holding along the way.

i can have a crack at the ECS event schema

Nice! yea lets figure it out.

See also https://wikitech.wikimedia.org/wiki/Event_Platform/Producer_Requirements#Events. We need to think about all the MUSTs there, but in general that is what we need. We might encounter a hiccup around ECS requirements and those MUST fields for events, but let's see. If ECS can just ignore those fields(?) or if they cause a problem in logstash, we could possibly just logstash filter them out?

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/45

Add a new schema that allows ingestion of ECS logs to the Event Platform

I've had a first go at this ECS schema in https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/45

I didn't know if it was best to put the first version under jsonschema/development/ to begin with, or straight into jsonschema/ - I went with the latter, for now.

Oh! Saw the MR before the Phab. Commented about development/ there too. It might be a good idea to prove that this works with this before we commit to its final name. OTOH, there probably isn't much iteration here since we are just copying an upstream schema (no bikeshedding needed! ;) ).

Very nice!

Change #1287374 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure rsyslog to forward 'dumps-http' messages to Kafka

https://gerrit.wikimedia.org/r/1287374

Change #1287407 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure nginx to log requests in ECS format to syslog

https://gerrit.wikimedia.org/r/1287407

Change #1295917 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] logstash: Consume the ECS dumps webrequest stream from Kafka

https://gerrit.wikimedia.org/r/1295917

Change #1295922 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/mediawiki-config@master] Declare the webrequest.dumps.dev0 stream in EventStreamConfig

https://gerrit.wikimedia.org/r/1295922

Nice! +1 to all the patches from me. Edited task description so we don't forget to remove the 'development/dev' naming.

Change #1287407 merged by Btullis:

[operations/puppet@production] Configure nginx to log requests in ECS format to syslog

https://gerrit.wikimedia.org/r/1287407

Change #1296535 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] dumps: web: Fix nginx ECS access log config so nginx can start

https://gerrit.wikimedia.org/r/1296535

Change #1296535 merged by Btullis:

[operations/puppet@production] dumps: web: Fix nginx ECS access log config so nginx can start

https://gerrit.wikimedia.org/r/1296535

The first part seems to be working. We have nginx emitting ECS formatted log entries to the journal with a syslog tag of dumps_http

btullis@clouddumps1002:/var/log/nginx$ journalctl -u nginx.service|tail -n 1|sed 's/^[^{]*://'|jq .
{
  "$schema": "/development/elastic/ecs/1.0.0",
  "meta": {
    "stream": "webrequest.dumps.dev0",
    "id": "9d3098f73bdd172315cebb5a870eb183",
    "dt": "2026-06-02T11:21:46+00:00",
    "domain": "dumps.wikimedia.org"
  },
  "dt": "2026-06-02T11:21:46+00:00",
  "@timestamp": "2026-06-02T11:21:46+00:00",
  "ecs": {
    "version": "1.11.0"
  },
  "event": {
    "dataset": "dumps.access",
    "kind": "event",
    "category": "web",
    "type": "access",
    "outcome": "success"
  },
  "http": {
    "request": {
      "method": "GET",
      "referrer": "",
      "bytes": 492
    },
    "response": {
      "status_code": 206,
      "body": {
        "bytes": 155705
      }
    },
    "version": "1.1"
  },
  "url": {
    "original": "/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2",
    "path": "/enwiki/latest/enwiki-latest-pages-articles-multistream.xml.bz2",
    "query": ""
  },
  "source": {
    "ip": "<redacted>"
  },
  "user_agent": {
    "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 (compatible; HubSpot DataHub; +https://www.hubspot.com)"
  },
  "destination": {
    "address": "dumps.wikimedia.org"
  }
}

I have been able to copy these messages into the ECS validation tool at. https://doc.wikimedia.org/ecs/validator.html and no errors were detected.

I'm aiming to roll out the rsyslog change (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1287374) at around 12:30 UTC today, disabling puppet across the fleet to allow me to test it on a couple of canary hosts, first.

Change #1287374 merged by Btullis:

[operations/puppet@production] Configure rsyslog to forward 'dumps_http' messages to Kafka

https://gerrit.wikimedia.org/r/1287374

Change #1296605 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] dumps: http: Stop prepending the hostname to the syslog events

https://gerrit.wikimedia.org/r/1296605

Change #1296607 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] kafka event platform logs - Strip the stray $!msg field

https://gerrit.wikimedia.org/r/1296607

Change #1296605 merged by Btullis:

[operations/puppet@production] dumps: http: Stop prepending the hostname to the syslog events

https://gerrit.wikimedia.org/r/1296605

Change #1296607 merged by Btullis:

[operations/puppet@production] kafka event platform logs - Strip the stray $!msg field

https://gerrit.wikimedia.org/r/1296607

I think that the nginx and rsyslog and kafka parts of this are now working.
Here is an example message from the kafka-jumbo cluster, which indicates that they have been produced to the kafka-logging cluaster and successfully mirrored to kafka-jumbo.

btullis@stat1008:~$ kafkacat -b kafka-jumbo1010.eqiad.wmnet -t eqiad.webrequest.dumps.dev0 -C -o -1 -c1|jq .
{
  "$schema": "/development/elastic/ecs/1.0.0",
  "meta": {
    "stream": "webrequest.dumps.dev0",
    "id": "620d7183d2c2d7693a901624da511f14",
    "dt": "2026-06-03T10:16:50+00:00",
    "domain": "dumps.wikimedia.org"
  },
  "dt": "2026-06-03T10:16:50+00:00",
  "@timestamp": "2026-06-03T10:16:50+00:00",
  "ecs": {
    "version": "1.11.0"
  },
  "event": {
    "dataset": "dumps.access",
    "kind": "event",
    "category": "web",
    "type": "access",
    "outcome": "success"
  },
  "http": {
    "request": {
      "method": "GET",
      "referrer": "",
      "bytes": 185
    },
    "response": {
      "status_code": 200,
      "body": {
        "bytes": 56940945
      }
    },
    "version": "1.1"
  },
  "url": {
    "original": "/other/pageviews/2026/2026-05/pageviews-20260515-050000.gz",
    "path": "/other/pageviews/2026/2026-05/pageviews-20260515-050000.gz",
    "query": ""
  },
  "source": {
    "ip": "<redacted>"
  },
  "user_agent": {
    "original": "google-xrawler"
  },
  "destination": {
    "address": "dumps.wikimedia.org"
  }
}

Change #1295917 merged by Btullis:

[operations/puppet@production] logstash: Consume the ECS dumps webrequest stream from Kafka

https://gerrit.wikimedia.org/r/1295917

Exciting! Every done a spiderpig deployment? I think https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1295922 can go out! (I am happy to do it for you if you prefer.)

The logstash integration is done, so we can now see the logs here: https://logstash.wikimedia.org/goto/3247aea974a35ec38baabcb4d716e022

Change #1295922 merged by jenkins-bot:

[operations/mediawiki-config@master] Declare the webrequest.dumps.dev0 stream in EventStreamConfig

https://gerrit.wikimedia.org/r/1295922

Mentioned in SAL (#wikimedia-operations) [2026-06-03T16:37:16Z] <btullis@deploy1003> Started scap sync-world: Backport for [[gerrit:1295922|Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-03T16:39:16Z] <btullis@deploy1003> btullis: Backport for [[gerrit:1295922|Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-06-03T16:44:32Z] <btullis@deploy1003> Finished scap sync-world: Backport for [[gerrit:1295922|Declare the webrequest.dumps.dev0 stream in EventStreamConfig (T291645 T425087)]] (duration: 07m 16s)

Event stream is configured:
https://stream-internal.wikimedia.org/v2/ui/#/%2F%3Fstreams=webrequest.dumps.dev0

But no data yet:

spark-sql (default)>  select count(*) from event.webrequest_dumps_dev0;
count(1)
0
Time taken: 8.135 seconds, Fetched 1 row(s)

Change #1297260 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion

https://gerrit.wikimedia.org/r/1297260

Change #1297260 merged by jenkins-bot:

[operations/mediawiki-config@master] EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion

https://gerrit.wikimedia.org/r/1297260

Mentioned in SAL (#wikimedia-operations) [2026-06-04T00:24:34Z] <otto@deploy1003> Started scap sync-world: Backport for [[gerrit:1297260|EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]]

Mentioned in SAL (#wikimedia-operations) [2026-06-04T00:26:40Z] <otto@deploy1003> otto: Backport for [[gerrit:1297260|EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Hm, Refine to Hive failing because the data does not match the schema. The data is produced directly to Kafka, so there is no validation step.

https://www.jsonschemavalidator.net/s/I4lrPRXY

@BTullis I think we can get away with the @timestamp additional property error, but it looks like there are a bunch of fields that should be strings that the schema has as an array, e.g. event.category, etc. Also some fields not defined (in the right place), e.g user_agent?

Mentioned in SAL (#wikimedia-operations) [2026-06-04T07:04:04Z] <otto@deploy1003> Finished scap sync-world: Backport for [[gerrit:1297260|EventStreamConfig - webrequest.dumps.dev0 - enable canary events for hive ingestion (T425087)]] (duration: 399m 30s)

Change #1297642 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] dumps: web: Make nginx ECS log_format conform to the Event Platform schema

https://gerrit.wikimedia.org/r/1297642

Change #1297642 merged by Btullis:

[operations/puppet@production] dumps: web: Make nginx ECS log_format conform to the Event Platform schema

https://gerrit.wikimedia.org/r/1297642

You have data in Hive! event.webrequest_dumps_dev0!

@KCVelaga_WMF we have data, are you unblocked?

Great! How much more testing to we have to do before promoting the schema from the development namespace, and who should make that call?

I believe we said that we wanted to enable canary events for the stream. Is that right?
Is it just a mediawiki-config change that we need, or anything else?

In T425087#11987887, @BTullis wrote:

Great! How much more testing to we have to do before promoting the schema from the development namespace, and who should make that call?

Thank you all for working on this. I can explore the data today or Monday and report back. Not sure about the schema decision though.

From my end as well, really thanks all for the efforts to get this set up! I just took a quick look at the data and it appears to meet our needs for the Wikidata dump metrics. I'm off for a few weeks starting tomorrow, but will look to adopt this new data soon after getting back 😊

wanted to enable canary events for the stream. Is that right?

Ya lets do it.

'canary_events_enabled' => true,
'destination_event_service' => 'eventgate-analytics'

And, once deployed, we need a rolling restart of eventgate-analytics. (...we should probably change this so restarts aren't needed here...)

How much more testing to we have to do before promoting the schema from the development namespace

I think since its now working, and we don't have to make any backwards incompatible changes, let's go for it!

If you like, you could version the stream, just in case we ever do have to do something backwards incompatible.

Also...it is time for stream name bikeshedding!

• Should all ECS streams have ecs somewhere in the name? Usually we don't necessarily couple schema names to stream names, but we can. I'm not sure. Hmmm.
• Without ^, I worry that webrequest.dumps will be conflated with the main webrequest_frontend or other webrequest streams. Yes these are all technically webrequests but hm, not sure.

Also, usually we try to name things more like namespace.thing_that_happened. What if we did:

dumps.webrequest.v1

?

I know that doesn't have ecs in the name, but I'm waffling on that. Thoughts? Ideas?

Anyway, I'm out next week, don't wait for approval from me to move forward with something!

In T425087#11987928, @KCVelaga_WMF wrote:

In T425087#11987887, @BTullis wrote:

Great! How much more testing to we have to do before promoting the schema from the development namespace, and who should make that call?

Thank you all for working on this. I can explore the data today or Monday and report back. Not sure about the schema decision though.

Sorry, I checked the data but for forgot to update back here. The data looks good to me, and is meets the requirements for further metrics.

Sorry, there are just too many different options here for me to be comfortable moving forward without a bit more guidance.

In T425087#11990062, @Ottomata wrote:

wanted to enable canary events for the stream. Is that right?

Ya lets do it.

Right, but do we do it before renaming the stream, or does it not matter one way or the other?

How much more testing to we have to do before promoting the schema from the development namespace

I think since its now working, and we don't have to make any backwards incompatible changes, let's go for it!

Same thing. Is it too early to promote it if we are still thinking about the following?

• renaming to include an ecs component
• renaming to add a version tag

Also...it is time for stream name bikeshedding!

• Should all ECS streams have ecs somewhere in the name? Usually we don't necessarily couple schema names to stream names, but we can. I'm not sure. Hmmm.
• Without ^, I worry that webrequest.dumps will be conflated with the main webrequest_frontend or other webrequest streams. Yes these are all technically webrequests but hm, not sure.

I can see both sides of the argument, but I don't feel like I'm the right person to make a strong recommendation.

Also, usually we try to name things more like namespace.thing_that_happened. What if we did:

dumps.webrequest.v1

Again, this sounds pretty good to me, but I'd rather be guided by someone else. There are certainly a lot of ECS fields that we haven't implemented, so even if nothing looks backwards-incompatible today, there's a good chance that we want to allow for accidentally making backwards-incompatible changes in the future.

So what about this as a stream name?
ecs.webrequest.dumps.v1

My guess is that there are potentially lots more sources of ecs format event (or log) records, which won't be generated by web requests.
We might need to add more fields from the ECS 1.11.0 spec that we deemed not relevant to this use-case, but might be relevant for other things, later on.

Hi @BTullis @Ottomata: can we move the schema out of development namespace to a permanent one, and also generate canary events?

Yes let's do it!

Oh I missed Ben's questions!

Right, but do we do it before renaming the stream, or does it not matter one way or the other?

Doesn't matter.

So what about this as a stream name?
ecs.webrequest.dumps.v1

I think...I'm convincing myself that the schema/format of the stream shouldn't influence the name of the stream. Mayyyybe if this was a log file we'd name it webrequest.dumps.v1.ecs.json? Or maybe if was an HTTP response we'd give it a Content-Type Header of application/json+ecs Or something?

But the actual name of the dataset...mmm nahh.

Also webrequest.dumps vs dumps.webrequest: I prefer the latter. This represents about a web (http) request to the dumps website. The verb is the '(web)request' part.

So, I think I most prefer dumps.webrequest.v1 if you don't mind. If you feel strongly about including ecs or having it another way let's jump in a bikeshedding session and work it out.

webrequest.dumps vs dumps.webrequest: I prefer the latter.

I could be convinced otherwise in this case though, since at WMF 'webrequest' is a pretty well used term. Perhaps namespacing should be backwards? I'm not sure!

webrequest.dumps vs dumps.webrequest

A related bikeshed for a 'webrequest' stream is in progress at https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1308656/1/wmf-config/ext-EventStreamConfig.php#2684. We should make the same choice for both.