Thanks for the ping @elukey!

A coupla things about the test-kitchen/standalone-event-system-success-rate SLI/SLO:

I noticed that xlab_sli_standalone_event_system_errors_total (numerator) isn't constrained to the product_metrics.web_base stream in the same way that xlab_sli_standalone_event_system_total (denominator). It should be! In noticing that, I also noticed that EventGate isn't validating the contents of the stream_name label prior to emitting the metrics. I raised T429799: [EventGate] Sanitize stream name in all metrics.

I had initially thought that xlab_sli_standalone_event_system_errors_total could be influenced by instrument and experiment code but that's not the case as it's only counting the HoistingError and MalformedHeaderError validation error types. However, it's possible that parts of an event could be invalid due to a system error, e.g. we introduce a bug in the Test Kitchen JS SDK which sets a bad value for the experiment field. These types of error should be part of the SLO definition.

The SLI was noticeably impacted between 06/15 and 06/23. There was a sudden, sharp decrease of events produced to the product_metrics.web_base stream at the same time on 06/15. I'll raise this with the Experiment Platform team and the Data Engineering team, who own and maintain the EventGate services.

In T421762#12041609, @phuedx wrote:

The SLI was noticeably impacted between 06/15 and 06/23. There was a sudden, sharp decrease of events produced to the product_metrics.web_base stream at the same time on 06/15. I'll raise this with the Experiment Platform team and the Data Engineering team, who own and maintain the EventGate services.

@mpopov investigated and determined that this was caused by the app install CentralNotice campaign:

SELECT
  date_trunc('hour', from_iso8601_timestamp(meta.dt)) AS dt,
  IF(experiment.coordinator IS NULL, instrument_name, experiment.enrolled) AS slug,
  IF(experiment.coordinator = 'default', 'A/B test', 'Instrument') AS source,
  COUNT(1) AS event_count
FROM product_metrics_web_base
WHERE year = 2026 AND month = 6
  AND ((day = 14 AND hour >= 20) OR (day = 15 AND hour < 4))
GROUP BY 1, 2, 3

So the "noticeable impact" I mentioned in my previous comment, is actually the SLI becoming more sensitive to the ongoing stream of hoisting errors (see T421152) as the overall event rate decreases.

In T421762#12041609, @phuedx wrote:

I had initially thought that xlab_sli_standalone_event_system_errors_total could be influenced by instrument and experiment code but that's not the case as it's only counting the HoistingError and MalformedHeaderError validation error types. However, it's possible that parts of an event could be invalid due to a system error, e.g. we introduce a bug in the Test Kitchen JS SDK which sets a bad value for the experiment field. These types of error should be part of the SLO definition.

In T427092: Run and synthetic A/A test that captures UA to investigate hoisting errors, we discovered that the majority of hoisting errors (the HoistingError error type) originate from a specific user agent, Bingbot. I've raised T429898: [EventGate] Add configurable UA denylist.