These events have the canary enabled, so they should have at least one event in every partition. event.mediawiki_revision_recommendation_create is missing the partition for datacenter=eqiad/year=2021/month=5/day=14/hour=1
Heya @EBernhardson, not having canary events in the refined data is expected: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/refine/TransformFunctions.scala#L50
I nonetheless think that partitions should be added even if the dataset is empty - ping @Ottomata on this.
Hmm, is this a new limitation? We first deployed this stream in february and no events came through for a few months, but we were getting hourly partitions anyway from the canary events. Git log suggests this is not new, so i'm not sure what changed.
Canary events are enabled for this stream. They are used to make create the partitions, but are filtered out of the refined event database. So, they will be in the stream and the raw data, but not in the final event table.
There was an outage of the canary events producer last week, fixed by https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/691232/.
https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/691236 will help, but we need to do T270138: produce_canary_events job should not fail if a schema is missing examples to keep that from happening in general.
A short outage (< than an hour) of canary events is fine, but yeah...if it is broken for more than an hour, a topic with no other data will not result in any Hive event table partitions created.
I'll try to bump the priority of T283084, this really shouldn't happen like this.
Oh, I think I forgot to update this because we never groomed it via Analytics tag.
I haven't done any work to backfill any partitions; there really were no events at all for that hour because ProduceCanaryEvents failed during that time.
ProduceCanaryEvents is now fixed so that a single stream produce failure won't cause all streams to have canary events failed to be produced, so hopefully this won't happen again (at least it shouldn't for that reason.)