Change 811212 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery/source@master] [WIP] Update refine to use Iceberg for event_sanitize

https://gerrit.wikimedia.org/r/811212

There are two issues currently in moving the sanitized data to iceberg:

• The meta.dt field is sanitized, preventing to use it as official timestamp for the event while it is more reliable than dt
• The meta.uuid field is sanitized, preventing to use it in case of merge (when rerunning a job, we want to overwrite existing data, which is done in iceberg using merge with an ID)

@Ottomata are there old eventlogging schemas not yet migrated to eventgate that would not have those two fields?

Ya there are some: T282131: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned. It looks a couple of the ones mentioned here do have some sanitize allow list in analytics/refinery.

Also all the ones in the 'unknown' tab of the Audit Spreadsheet, which is mostly Mobile Apps. IIRC, they are not migrating these, but will be decommissioning them and making new ones based on metrics platform. Don't know the timeline. I do see MobileWikiApp schemas mentioned in the sanitize allowlist too.

Eventually, a schema will be migrated or decommissioned. Once migrated, they will have those fields. Right now, though, ya there are some that don't.

Problem statement for this use-case here: https://docs.google.com/document/d/1HVO4m8JG5mrYX9ltdvJdt8N3QVscspdNVbpUY2kKtyo/edit