The last survey impression seen on Hive in the QuickSurveyInitiation Eventlogging schema is exactly at 2020-02-07T16:59:59Z
This timing, just before the hour, is suspicious to say the least! We stopped getting impressions on all wikis at once.
I don't see anything related in the SAL around that time.
Seems like the delay of data to Hive is expected, generally more than 2 hours. Let's see if the data turns up later.
See intake on kafka: https://grafana.wikimedia.org/d/000000018/eventlogging-schema?orgId=1&from=now-6h&to=now&var-schema=QuickSurveyInitiation
There is a drop that coincides with a deploy or outage of some sort
Actually, the issue is the alert setup I think. That's what I was trying to figure out here. I think the alert was firing because it's looking at 24 hour aggregates and if we have less than 24 hours of data for the current day, to the alert system it looks like the partial data for the day is below the threshold.
It was indeed the alert definition. I've changed them to use a different strategy, you should take a look.
