I think array fields should only be null when they aren't relevant to the row -- e.g., user_ fields in a revision create event. Otherwise, it's really easy to make logical errors in your queries and accidentally drop rows that are null. Two concrete examples:
- For edits, there's a revision_tags field that is array<string> and contains any edit tags associated with the edit. When there are no edit tags applied (not common now but something that was a lot more common 5-10 years ago before we started tagging everything with the editor/platform it came from), that field is null. If you naively do something like AND NOT ARRAY_CONTAINS('revision_tags', 'mw-reverted') to filter out reverted edits, you also end up filtering out any edit that lacks an edit tag too because ARRAY_CONTAINS returns a null when the array is null and the NOT operator doesn't change that. Instead, folks have to do something like AND NOT COALESCE(ARRAY_CONTAINS('revision_tags', 'mw-reverted'), False)
- Similar example but different operator. Say you want to remove blocked editors from your query. You might add something like SIZE(event_user_blocks) = 0 but if event_user_blocks is null, then SIZE returns -1 and so you actually need SIZE(event_user_blocks) < 1 if you want to be inclusive of those cases (or NOT SIZE(event_user_blocks ) > 0. Further complications if you happen to use PySpark's array_size function as that returns null for null arrays.
Fields where this might apply -- I haven't checked all for presence of unexpected null values but I suspect most if not all have it:
- event_user_blocks_historical
- event_user_blocks
- event_user_groups_historical
- event_user_groups
- event_user_is_bot_by_historical
- event_user_is_bot_by
- user_blocks_historical
- user_blocks
- user_groups_historical
- user_groups
- user_is_bot_by_historical
- user_is_bot_by
- revision_deleted_parts
- revision_tags