EventGate: Add Prometheus metric for hoisting errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phuedx
	Jul 8 2025, 8:37 AM

Project Tags

Referenced Files

	F65716716: MalformedHeaderError.png
	Aug 5 2025, 1:02 PM

	F65716715: HoistingError.png
	Aug 5 2025, 1:02 PM

	F65690075: Screenshot 2025-07-29 at 5.51.51 PM.png
	Jul 29 2025, 11:34 PM

	F65690029: Screenshot 2025-07-29 at 5.29.27 PM.png
	Jul 29 2025, 11:34 PM

Subscribers

Description

Background

So-called "hoisting errors" were added to EventGate in T391959: FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2. A hoisting error occurs when either the CDN sends an un-parseable X-Experiment-Enrollments header or when an experiment-related analytics event contains invalid experiment enrollment information.

Hoisting errors are somewhat like validation errors in that they are thrown during the event validation process and they are caught and logged. Validation errors, however, are counted in Prometheus.

The Experiment Platform team request that hoisting errors are counted in Prometheus like validation errors. This would enable us to define and monitor our Experimentation Lab/Request Class 2 SLI in Pyrra.

AC

Prometheus metric for HoistingError exists.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	eventgate-analytics-external - bump to 1.19.0 to pick up recent ExP changes	Ottomata	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Export ContextualError so it can be extended and used	repos/data-engineering/eventgate!6	otto	T398922_ContextualError_export	master

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	dr0ptp4kt	T400194 Adam's Hypothesis drafting for FY 2025-2026
Resolved	dr0ptp4kt	T398229 FY25-26 SDS2.1.3 Reliability - Production Monitoring
Resolved	Milimetric	T382153 Define Product-Level Service Level Objectives (SLOs) for Experimentation Lab
Resolved	dr0ptp4kt	T398869 Create Pyrra SLOs for xLab
Resolved	Ottomata	T398922 EventGate: Add Prometheus metric for hoisting errors

Event Timeline

phuedx created this task.Jul 8 2025, 8:37 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 8 2025, 8:37 AM

phuedx added a parent task: T398229: FY25-26 SDS2.1.3 Reliability - Production Monitoring.Jul 8 2025, 9:03 AM

Maintenance_bot added a project: Data-Engineering.Jul 8 2025, 9:30 AM

phuedx added a parent task: T398869: Create Pyrra SLOs for xLab.Jul 9 2025, 8:23 AM

dr0ptp4kt subscribed.Jul 9 2025, 4:54 PM

Milimetric moved this task from Incoming to Radar on the Test Kitchen board.Jul 14 2025, 3:30 PM

Ottomata subscribed.Jul 15 2025, 5:34 PM

Vgutierrez subscribed.Jul 16 2025, 2:27 PM

Dan and I put this in DE Next Up. @phuedx what is the priority of this? Do you have a deadline of when you'd like it done and deployed?

Since @tchin worked on Hoisting errors, he could probably tackle this.

Ahoelzl assigned this task to tchin.Jul 16 2025, 6:19 PM

Ahoelzl edited projects, added: Data-Engineering (Q4 2025 April 1st - June 30th); removed: Data-Engineering.

I was hoping to load stuff into Pyrra by around the end of July.

Is it viable to add this functionality over the course of the next two weeks? If not, I'd also be happy to take a run at it if I could get a code reviewer and someone to be my gentle guide for any deployment.

I can appreciate, however, it may make more sense for a core maintainer of EventGate / EventGate Wikimedia to do this instead of me (and I can code review, test, etc.). Happy to be a deployer for it even if coded by someone else (doing an EventGate deployment is on my TODO list).

If you want to drive it, be my guest and I can help you out if needed. Or we can pair program together. Whichever you prefer

Ottomata moved this task from Backlog to Components on the Event-Platform board.Jul 21 2025, 8:26 PM

I'm in this code for another task so I can do this.

Because HoistingError extends from ValidationError, I think they are currently being counted as ValidationErrors in prometheus.

https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/blob/master/eventgate-wikimedia.js#L247

https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/blob/master/eventgate-wikimedia.js#L322

In makeMapToErrorEvent - update to /error/2.1.0 and don't set meta.dt, I'm updating this function to set dt fields properly, and also populating the error_type field with the name of the error.

Shall we just add a new error_type label to the existent eventgate_validation_errors_total prometheus metric? Or should we make a new metric?

I think I'm inclined to make a new metric. I think that perhaps it would be better if HoistingError did not subclass ValidationError, but instead subclass ContextualError from EventGate lib upstream. IIUC, the only reason this was not done is that EventGate didn't export ContextualError. We can make it do so!

So, I propose

Modify EventGate so it exports ContextualError
Change HoistingError to extend from ContextualError
Emit a new eventgate_hoisting_errors_total metric.

Question: @dr0ptp4kt, Do you need hoisting errors to be emitted as error events? ValidationErrors are currently ( and I think HoistingErrors are too?). These would be emitted as events to a stream (we might create a new stream?) and contain a copy of the stringified raw event that caused the error.

Ottomata triaged this task as High priority.Jul 22 2025, 8:17 PM

Ottomata moved this task from Next Up to In progress on the Data-Engineering (Q4 2025 April 1st - June 30th) board.

I think I'm inclined to make a new metric. I think that perhaps it would be better if HoistingError did not subclass ValidationError, but instead subclass ContextualError from EventGate lib upstream.

I think that modelling is more representative, yes.

Question: @dr0ptp4kt, Do you need hoisting errors to be emitted as error events? ValidationErrors are currently ( and I think HoistingErrors are too?).

I stumbled across a Slack thread, which included a link to this task that Experiment Platform created requesting precisely this: T396359: EventGate: Log unparseable X-Experiment-Enrollments headers to an error stream

These would be emitted as events to a stream (we might create a new stream?) and contain a copy of the stringified raw event that caused the error.

Would that event include the header that caused the hoisting error?

T396359, Right okay!

Would that event include the header that caused the hoisting error?

Hm, let's try.

Ottomata mentioned this in T396359: EventGate: Log unparseable X-Experiment-Enrollments headers to an error stream.Jul 23 2025, 3:36 PM

Same as @phuedx , a new metric sounds good to me. Adding another label would be fine, but this circumstance is more special, and I can imagine how we may want to create custom labels for the new metric in due time, too, and avoid overindulgence with the existing metric's labels.

As for the refactoring for subclassing, not a blocker, although I think I get the rationale. The original discussion about that approach implied that the failure case here might not constitute a schema validation error but rather something else. I would say it is in a somewhat in-between state, in that there is latent schema meaning when we see a value that cannot be present in the event payload (and yet, I think it's fine that they're handled in the same concrete error handling routine, as opposed to demarcated artificially). But, to the point, it's not at the level of general schema validation based on typing or deterministic and correct JSONSchema regex-enforceability, so I can see the case!

I was unclear on the historical addition of ContextualError looking between https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/commit/2ed78ebd05d070eaeb50f64ee0cfc7e11bcdc620 and https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/commit/74a3c0d249c5b803ea553a71cb17aae5223180c2 (which is about other formalities of schema validation) and a naive Phabricator search, but I think from the code review comments on the original hoisting part and the fix update they're indicative of the true intent for ContextualError. So, it sounds sensible!

For working toward the target for SLO piece, the new metrics is the primary thing, of course.

I'll hop over to T396359: EventGate: Log unparseable X-Experiment-Enrollments headers to an error stream for commentary about the error stream-side mechanics.

Ahoelzl edited projects, added: Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed: Data-Engineering (Q4 2025 April 1st - June 30th).Jul 25 2025, 10:56 PM

Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.

otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/6

Export ContextualError so it can be extended and used

otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/merge_requests/6

Export ContextualError so it can be extended and used

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 3:31 PM

Change HoistingError to extend from ContextualError

Hm, When extended from ValidationError, the HoistingErrors were produced as events in the eventgate-analytics-external.error.validation stream. Do you want/need this? And, if we are considering HoistingError != ValidationError, should we make HoistingErrors produced to their own stream?

Or, perhaps: we should refactor the 'validation' error streams to a more generic eventgate-analytics-external.error stream, and then use the error_type field added in 2.1.0 to differentiate?

EDIT:

Oh, I asked this already and Sam said:

this task that Experiment Platform created requesting precisely this: T396359: EventGate: Log unparseable X-Experiment-Enrollments headers to a distinct error stream

Reading that task, that task is asking for MalformedHeaderError, not HoistingError to be emitted to the error event stream.

Shall we just add a new error_type label to the existent eventgate_validation_errors_total prometheus metric? Or should we make a new metric?

Hm, another consideration: EventGate actually throws quite a few other kinds of errors: https://gitlab.wikimedia.org/repos/data-engineering/eventgate/-/blob/master/lib/error.js?ref_type=heads.

eventgate-wikimedia has a version of a UnauthorizedSchemaForStreamError that is thrown during the validation step, usually because of misconfigured streams.

If we made the error_type a label and used the same metric for all of these, I think eventgate would be better instrumented overall. We'd be able to chart and/or alert on these at will.

So, my current preference is:

Keep the eventgate_validation_errors_total metric (or rename it?) and add error_type or as a label.
Keep the eventgate-analytics-external.error.validation stream (or rename it?) and populate the error_type field (already done).

While it would be nice to rename the metric and stream to be more generic (e.g. eventgate_event_errors_total metric and/or eventgate-analytics-external.event_error stream(s)), doing so will be a bit more work: stream config declaration, eventgate-wikimedia helmfile config change, grafana dashboard changes, etc.

So, let's keep the names the same for now, and I'll add some comments in the code about the history and reasons for the name, and the desire to rename, perhaps one day.

I think this approach will also allow us to be more flexible for T396359 too. Instead of distinct new streams, we just forward the error events to the existent stream with error_type as the differentiator.

otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/21

Add metric for experiment hoisting errors

Alright! https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/21 is ready for review!

In T398922#11040416, @Ottomata wrote:

If we made the error_type a label and used the same metric for all of these, I think eventgate would be better instrumented overall. We'd be able to chart and/or alert on these at will.

Super cool!

otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventgate-wikimedia/-/merge_requests/21

Add metric for experiment hoisting errors

Deployed and tested code in beta, but I can't directly test HoistingErrors there.

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2025, 4:31 PM

Ottomata moved this task from In progress to Ready to Deploy on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Jul 29 2025, 7:20 PM

Varnish will not allow one to fake out one's X-Experiment-Enrollments header from the outside, so one has to get to the interior network and hit the internal true route if wanting to emulate the header enrichment that Varnish does and induce EventGate error handling normally suited only for Edge Uniques-oriented processing.

The Beta Cluster route rewriting rule is shown in WMCS's deployment-prep/deployment-cache-text.yaml and the prod one is shown in the Puppet trafficserver/backend.yaml.

Now, here's what I see in Beta Cluster when contrasting a well-formed JSON request with the subject_id of awaiting versus having an invalid subject_id...where in both cases the X-Experiment-Enrollments is a well-formed header.

$ ssh deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud

deployment-deploy04:~$ cat beta.good.bash
curl -i -s -H "X-Experiment-Enrollments: gastropub=proteinchips/1234567890abcdefghiJKz;" -H "Content-Type: application/json"  -X POST http://deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud:8492/v1/events -d @beta.good.json

deployment-deploy04:~$ cat beta.good.json
{
  "$schema": "/analytics/product_metrics/web/base/1.4.2",
  "dt": "2025-07-29T11:12:11.111Z",
  "meta": {
    "stream": "product_metrics.web_base"
  },
  "action": "clicked",
  "experiment": {
    "enrolled": "gastropub",
    "assigned": "proteinchips",
    "subject_id": "awaiting",
    "sampling_unit": "edge-unique",
    "coordinator": "forced",
    "other_assigned":
      {
        "ice_cream_cone": "yes"
      }
  },
  "agent" : {
    "client_platform": "mediawiki_js",
    "client_platform_family": "desktop_browser",
    "release_status": "dev"
  }
}

deployment-deploy04:~$ bash beta.good.bash
HTTP/1.1 201 All 1 out of 1 events were accepted.
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
access-control-allow-methods: post
Date: Tue, 29 Jul 2025 21:48:29 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

deployment-deploy04:~$ cat beta.bad.bash
curl -i -s -H "X-Experiment-Enrollments: gastropub=proteinchips/1234567890abcdefghiJKz;" -H "Content-Type: application/json"  -X POST http://deployment-eventgate-4.deployment-prep.eqiad1.wikimedia.cloud:8492/v1/events -d @beta.bad.json
dr0ptp4kt@deployment-deploy04:~$ cat beta.bad.json
{
  "$schema": "/analytics/product_metrics/web/base/1.4.2",
  "dt": "2025-07-29T11:12:11.111Z",
  "meta": {
    "stream": "product_metrics.web_base"
  },
  "action": "clicked",
  "experiment": {
    "enrolled": "gastropub",
    "assigned": "proteinchips",
    "subject_id": "BAD!!!awaiting!!!BAD",
    "sampling_unit": "edge-unique",
    "coordinator": "forced",
    "other_assigned":
      {
        "ice_cream_cone": "yes"
      }
  },
  "agent" : {
    "client_platform": "mediawiki_js",
    "client_platform_family": "desktop_browser",
    "release_status": "dev"
  }
}

(And here was visual evidence that the same event sent later came through.)

Screenshot 2025-07-29 at 5.29.27 PM.png (3,714×584 px, 226 KB)

(Now continuing to inducing an error...)

deployment-deploy04:~$ bash beta.bad.bash
HTTP/1.1 500 1 out of 1 events had failures and were not accepted. (0 invalid and 1 errored).
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
access-control-allow-methods: post
Content-Type: application/json; charset=utf-8
Content-Length: 797
Vary: Accept-Encoding
Date: Tue, 29 Jul 2025 21:49:05 GMT
Connection: keep-alive
Keep-Alive: timeout=5

{"invalid":[],"error":[{"status":"error","event":{"$schema":"/analytics/product_metrics/web/base/1.4.2","dt":"2025-07-29T11:12:11.111Z","meta":{"stream":"product_metrics.web_base","id":"5355ad3a-8e8a-41c1-a96a-8b5baccd2b45","dt":"2025-07-29T21:49:05.231Z","request_id":"d8edb6e0-6cc5-11f0-b0da-dd91866a4461"},"action":"clicked","experiment":{"enrolled":"gastropub","assigned":"proteinchips","subject_id":"BAD!!!awaiting!!!BAD","sampling_unit":"edge-unique","coordinator":"forced","other_assigned":{"ice_cream_cone":"yes"}},"agent":{"client_platform":"mediawiki_js","client_platform_family":"desktop_browser","release_status":"dev"}},"context":{"message":"Could not hoist data into experiment.subject_id: Event is enrolled in experiments but 'experiement.subject_id' is not a valid subject_id."}}]

I see the error with a manual kafkcat from deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud:9092 (note this example payload was one I sent later) for the Beta Cluster...

$ ssh deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud:9092
deployment-kafka-jumbo-9:~$ kafkacat -C -b deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud:9092 -t eqiad.eventgate-analytics-external.error.validation
...
{"meta":{"id":"43355740-0a5f-4eb3-9609-bf60b20afeb5","uri":"unknown","domain":"unknown","request_id":"ac528f80-6cce-11f0-b0da-dd91866a4461","stream":"eventgate-analytics-external.error.validation","dt":"2025-07-29T22:52:15.873Z"},"dt":"2025-07-29T22:52:15.871Z","emitter_id":"eventgate-analytics-external","raw_event":"{\"$schema\":\"/analytics/product_metrics/web/base/1.4.2\",\"dt\":\"2025-07-29T11:12:11.111Z\",\"meta\":{\"stream\":\"product_metrics.web_base\",\"id\":\"81ab61bf-f352-4297-834c-12c9bb436731\",\"dt\":\"2025-07-29T22:52:15.865Z\",\"request_id\":\"ac528f80-6cce-11f0-b0da-dd91866a4461\"},\"action\":\"clicked\",\"experiment\":{\"enrolled\":\"gastropub\",\"assigned\":\"proteinchips\",\"subject_id\":\"BAD!!!awaiting!!!BAD\",\"sampling_unit\":\"edge-unique\",\"coordinator\":\"forced\",\"other_assigned\":{\"ice_cream_cone\":\"yes\"}},\"agent\":{\"client_platform\":\"mediawiki_js\",\"client_platform_family\":\"desktop_browser\",\"release_status\":\"dev\"}}","errored_schema_uri":"/analytics/product_metrics/web/base/1.4.2","errored_stream_name":"product_metrics.web_base","message":"Could not hoist data into experiment.subject_id: Event is enrolled in experiments but 'experiement.subject_id' is not a valid subject_id.","$schema":"/error/2.1.0","error_type":"HoistingError"}

...although it didn't roll through the web UI that I saw. I'm gathering maybe that's not wired up for EventStream in Beta Cluster or maybe it's a symptom of something else. I was wondering if the 500 response managed to get the event onto the Kafka topic but somehow didn't relay down to EventStreams...or maybe that's just how it is in Beta Cluster.

Screenshot 2025-07-29 at 5.51.51 PM.png (3,714×584 px, 163 KB)

I'm unsure if application errors are able to be shown in Logstash in the Beta Cluster; I don't think I see them there, but maybe someone has a trick?

Starting one's scrolling from T391959#10806340 there are examples of hitting the Varnish edge in the Beta Cluster and in production. When we were doing that we were, for the purposes of testing, allowing X-Experiment-Enrollments headers to be set by the client from the wild so they would pass through and we could verify EventGate was functioning as expected. But this was ahead of activating Edge Uniques for real. Once Edge Uniques was activated for real, we started disallowing clients from the wild sending X-Experiment-Enrollments, only ever allowing Varnish to set the header.

And so, to verify in production upon its deployment there, you'd want to repeat the same thing, but do it from any host that has production reachability to https://eventgate-analytics-external.discovery.wmnet:4692/v1/events (e.g., via a shell at deployment.wikimedia.org) and altering the URLs in the command files for that URL, accordingly.

@Ottomata happen to know if there's an easy way to see the application layer errors from EventGate in Beta Cluster in Logstash in BetaCluster, or as evidenced by some Beta Cluster logging topic perhaps (e.g., one from deployment-kafka-logging01.deployment-prep.eqiad1.wikimedia.cloud:9092 )?

Anyway, hope this provides some help on how to verify some things in Beta Cluster.

Change #1174745 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-analytics-external - bump to 1.19.0 to pick up recent ExP changes

https://gerrit.wikimedia.org/r/1174745

gerritbot added a project: Patch-For-Review.Jul 31 2025, 3:31 PM

Change #1174745 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate-analytics-external - bump to 1.19.0 to pick up recent ExP changes

https://gerrit.wikimedia.org/r/1174745

Mentioned in SAL (#wikimedia-operations) [2025-07-31T15:59:57Z] <ottomata> deploying eventgate-analytics external for T398922 and T396359

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2025, 4:37 PM

Deployed eventgate-analytics-external to prod. HoistingError is now an error type. Added to grafana panel:

https://grafana.wikimedia.org/goto/v9D5E-wHR?orgId=1

Ottomata updated the task description. (Show Details)Jul 31 2025, 7:08 PM

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.

And, as expected, it's working in prod fine for HoistingError and MalformedHeaderError related error events.

# session in terminal 1
$ ssh deployment.eqiad.wmnet

deploy1003 ~$ cat prod.bad.HoistingError.bash
curl -i -s -H "X-Experiment-Enrollments: gastropub=proteinchips/1234567890abcdefghiJKz;" -H "Content-Type: application/json"  -X POST https://eventgate-analytics-external.discovery.wmnet:4692/v1/events -d @prod.bad.HoistingError.json
deploy1003 ~$ cat prod.bad.HoistingError.json
{
  "$schema": "/analytics/product_metrics/web/base/1.4.2",
  "dt": "2025-07-29T11:12:11.111Z",
  "meta": {
    "stream": "product_metrics.web_base"
  },
  "action": "clicked",
  "experiment": {
    "enrolled": "gastropub",
    "assigned": "proteinchips",
    "subject_id": "BAD!!!awaiting!!!BAD",
    "sampling_unit": "edge-unique",
    "coordinator": "forced",
    "other_assigned":
      {
        "ice_cream_cone": "yes"
      }
  },
  "agent" : {
    "client_platform": "mediawiki_js",
    "client_platform_family": "desktop_browser",
    "release_status": "dev"
  }
}
deploy1003 ~$ bash prod.bad.HoistingError.bash 
HTTP/1.1 500 Internal Server Error
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src 'none'; img-src 'none'; style-src 'none'; base-uri 'self'; frame-ancestors 'self'
content-type: application/json; charset=utf-8
content-length: 911
vary: Accept-Encoding
date: Tue, 05 Aug 2025 12:46:00 GMT
x-envoy-upstream-service-time: 1
server: production-tls

{"invalid":[],"error":[{"status":"error","event":{"$schema":"/analytics/product_metrics/web/base/1.4.2","dt":"2025-07-29T11:12:11.111Z","meta":{"stream":"product_metrics.web_base","id":"81eb7a22-d38e-4744-9c53-b865e2dad50b","dt":"2025-08-05T12:46:00.401Z","request_id":"acceb06d-a124-4def-82aa-d2407fd89f57"},"action":"clicked","experiment":{"enrolled":"gastropub","assigned":"proteinchips","subject_id":"BAD!!!awaiting!!!BAD","sampling_unit":"edge-unique","coordinator":"forced","other_assigned":{"ice_cream_cone":"yes"}},"agent":{"client_platform":"mediawiki_js","client_platform_family":"desktop_browser","release_status":"dev"}},"context":{"message":"Could not hoist data into experiment.subject_id for event event of schema at /analytics/product_metrics/web/base/1.4.2 destined to stream product_metrics.web_base: Event is enrolled in experiments but 'experiment.subject_id' is not a valid subject_id."}}]}

# different session in terminal 2, this was actually an older event done several minutes earlier, but same thing

$ ssh stat1010.eqiad.wmnet

stat1010:~$ kafkacat -C -b kafka-jumbo1007.eqiad.wmnet:9092 -t eqiad.eventgate-analytics-external.error.validation -o end | grep gastropub

{"meta":{"id":"fb6176f7-dc1b-4a93-b403-6b1d5dc24365","uri":"unknown","domain":"unknown","request_id":"8ba005f9-198d-45f9-924a-e7591488755d","stream":"eventgate-analytics-external.error.validation","dt":"2025-08-05T12:21:51.434Z"},"dt":"2025-08-05T12:21:51.434Z","emitter_id":"eventgate-production","raw_event":"{\"$schema\":\"/analytics/product_metrics/web/base/1.4.2\",\"dt\":\"2025-07-29T11:12:11.111Z\",\"meta\":{\"stream\":\"product_metrics.web_base\",\"id\":\"e6807ff0-478e-4ed8-8c27-645f893b4171\",\"dt\":\"2025-08-05T12:21:51.432Z\",\"request_id\":\"8ba005f9-198d-45f9-924a-e7591488755d\"},\"action\":\"clicked\",\"experiment\":{\"enrolled\":\"gastropub\",\"assigned\":\"proteinchips\",\"subject_id\":\"BAD!!!awaiting!!!BAD\",\"sampling_unit\":\"edge-unique\",\"coordinator\":\"forced\",\"other_assigned\":{\"ice_cream_cone\":\"yes\"}},\"agent\":{\"client_platform\":\"mediawiki_js\",\"client_platform_family\":\"desktop_browser\",\"release_status\":\"dev\"}}","errored_schema_uri":"/analytics/product_metrics/web/base/1.4.2","errored_stream_name":"product_metrics.web_base","message":"Could not hoist data into experiment.subject_id for event event of schema at /analytics/product_metrics/web/base/1.4.2 destined to stream product_metrics.web_base: Event is enrolled in experiments but 'experiment.subject_id' is not a valid subject_id.","$schema":"/error/2.1.0","error_type":"HoistingError"}

...

# terminal in session 1

deploy1003 ~$ cat prod.bad.MalformedHeaderError.bash 
curl -i -s -H "X-Experiment-Enrollments: too=short/8675309;" -H "Content-Type: application/json"  -X POST https://eventgate-analytics-external.discovery.wmnet:4692/v1/events -d @prod.bad.MalformedHeaderErrorActuallyOkayJustAdjustedForGrepping.json

deploy1003 ~$ cat prod.bad.MalformedHeaderErrorActuallyOkayJustAdjustedForGrepping.json 
{
  "$schema": "/analytics/product_metrics/web/base/1.4.2",
  "dt": "2025-08-05T12:13:14.150Z",
  "meta": {
    "stream": "product_metrics.web_base"
  },
  "action": "clicked",
  "experiment": {
    "enrolled": "gastroprobs",
    "assigned": "proteinchips",
    "subject_id": "awaiting",
    "sampling_unit": "edge-unique",
    "coordinator": "forced",
    "other_assigned":
      {
        "ice_cream_cone": "yes"
      }
  },
  "agent" : {
    "client_platform": "mediawiki_js",
    "client_platform_family": "desktop_browser",
    "release_status": "dev"
  }
}

deploy1003 ~$ bash prod.bad.MalformedHeaderError.bash 
HTTP/1.1 500 Internal Server Error
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
content-security-policy: default-src 'self'; object-src 'none'; media-src 'none'; img-src 'none'; style-src 'none'; base-uri 'self'; frame-ancestors 'self'
content-type: application/json; charset=utf-8
content-length: 694
vary: Accept-Encoding
date: Tue, 05 Aug 2025 12:51:53 GMT
x-envoy-upstream-service-time: 1
server: production-tls

{"invalid":[],"error":[{"status":"error","event":{"$schema":"/analytics/product_metrics/web/base/1.4.2","dt":"2025-08-05T12:13:14.150Z","meta":{"stream":"product_metrics.web_base","id":"49693dc3-f677-4274-83c8-25d291131d9c","dt":"2025-08-05T12:51:53.389Z","request_id":"0962b7d1-99b0-485b-b3d7-198458405390"},"action":"clicked","experiment":{"enrolled":"gastroprobs","assigned":"proteinchips","subject_id":"awaiting","sampling_unit":"edge-unique","coordinator":"forced","other_assigned":{"ice_cream_cone":"yes"}},"agent":{"client_platform":"mediawiki_js","client_platform_family":"desktop_browser","release_status":"dev"}},"context":{"message":"x-experiment-enrollments header is malformed"}}]}


# terminal 2, note this is actually an older event from several minutes earlier, but same idea

stat1010:~$ kafkacat -C -b kafka-jumbo1007.eqiad.wmnet:9092 -t eqiad.eventgate-analytics-external.error.validation -o end | grep gastroprobs

{"meta":{"id":"c2224392-c94f-451f-ae35-43fd98808005","uri":"unknown","domain":"unknown","request_id":"f61bd8e4-4e52-4e61-91d5-f960edd999d6","stream":"eventgate-analytics-external.error.validation","dt":"2025-08-05T12:37:47.588Z"},"dt":"2025-08-05T12:37:47.587Z","emitter_id":"eventgate-production","raw_event":"{\"$schema\":\"/analytics/product_metrics/web/base/1.4.2\",\"dt\":\"2025-08-05T12:13:14.150Z\",\"meta\":{\"stream\":\"product_metrics.web_base\",\"id\":\"56eccfe5-7e34-408e-9517-4c9906b2e424\",\"dt\":\"2025-08-05T12:37:47.584Z\",\"request_id\":\"f61bd8e4-4e52-4e61-91d5-f960edd999d6\"},\"action\":\"clicked\",\"experiment\":{\"enrolled\":\"gastroprobs\",\"assigned\":\"proteinchips\",\"subject_id\":\"awaiting\",\"sampling_unit\":\"edge-unique\",\"coordinator\":\"forced\",\"other_assigned\":{\"ice_cream_cone\":\"yes\"}},\"agent\":{\"client_platform\":\"mediawiki_js\",\"client_platform_family\":\"desktop_browser\",\"release_status\":\"dev\"}}","errored_schema_uri":"/analytics/product_metrics/web/base/1.4.2","errored_stream_name":"product_metrics.web_base","message":"x-experiment-enrollments header is malformed","$schema":"/error/2.1.0","error_type":"MalformedHeaderError"}

And we see this at https://thanos.wikimedia.org/ after inducing multiple errors of each type.

HoistingError...

MalformedHeaderError...

Marking this task as resolved. Thank you!

EventGate: Add Prometheus metric for hoisting errorsClosed, ResolvedPublicActions