The SLO on the Test Kitchen EventGate side of things is currently projected to be at risk, but errors do, as @cjming noted on chat seem to have tapered off considerably (we saw initial error budget alerting via T412448: ErrorBudgetBurn (sticky-headers) but as discussed there a backport was pushed to fix the main problem).

I think we can give it some time and see if it recovers.

A hastily constructed query suggests the overall error rate higher than we'd want to tolerate on a longer term basis, causing sub-99.75% success versus the hoped for 99.9% success.

But, keep in mind the errors are measured via a centralized counter, whereas the non-errors are counted via product_metrics.web_base. This is intentional and known, as data_platform.pp notes:

# For EventGate events, system errors (for numerator before complement)
# may be from arbitrary streams via schema fragments, but in general the
# desired destination stream will be product_metrics.web_base (for
# denominator). This means the error ratio may exaggerate SLO misses, which is
# okay in the context of the event system success rate measurement.

We can see in a different query for the originally flapping stream that the error rate does exist on that stream.

It's just that this experiment in T412146: Launch Mobile Expanded Sections on non-English wikis is running on some sites with reasonably bigger traffic at 10% sampling and so this is appearing to seep through to the global metric we're monitoring.

Indeed, if we recalculate so as to exclude the sticky header schema, mediawiki.product_metrics.readerexperiments_stickyheaders, for the errors, then we're doing okay.

However, we do have some errors occurring elsewhere that are resulting in us being a bit close to 99.9% success for comfort, so we may need to come up with a way to handle that in terms of adjusting recording rules / adding new Prometheus metric label emission or possibly handling even closer to the fundamental root of the problem (in or near concerns regarding cookie expiration, for example, which is a knotty problem).

sum by (stream, error_type) (rate(eventgate_validation_errors_total{service="eventgate-analytics-external", prometheus="k8s", error_type="HoistingError"}[1d]))

{error_type="HoistingError", stream="mediawiki.product_metrics.readerexperiments_imagebrowsing"}
0.00003474153218396851
{error_type="HoistingError", stream="mediawiki.product_metrics.readerexperiments_stickyheaders"}
0.04143786160336746
{error_type="HoistingError", stream="product_metrics.web_base"}
0.007747897720029626

# MalformedHeaderError showed no errors for the previous day