PS: patch is at https://gerrit.wikimedia.org/r/490514 (seems @gerritbot is lagging a bit currently)

NB: The names of these sample field names are spelled with underscores in Hive (e.g. page_issues_b_sample, see below) but with dashes in the schema page (e.g. page-issues-b_sample ). Which version does the whitelist require?

hive (default)> DESCRIBE event.readingdepth;
OK
col_name	data_type	comment
dt                  	string              	                    
event               	struct<action:string,domInteractiveTime:bigint,firstPaintTime:bigint,isAnon:boolean,namespaceId:bigint,pageTitle:string,pageToken:string,sessionToken:string,skin:string,totalLength:bigint,visibleLength:bigint,default_sample:boolean,page_issues_a_sample:boolean,page_issues_b_sample:boolean>	                    
recvfrom            	string
...

@Jdrewniak points out that in https://github.com/wikimedia/mediawiki-skins-MinervaNeue/blob/f07985c6dee5106da8f381a47214e7349fcd147e/resources/skins.minerva.scripts/pageIssuesLogger.js#L65 the spelling is still page-issues-b_sample/ page-issues-a_sample (i.e. like on the schema page, not like in Hive).

Blocked on code review and an answer to T216096#4953210 from someone familiar with the whole EL pipeline and the purging mechanism (@mforns?).

@Tbayer Thanks for spotting this!

Even if the events arrive to HDFS with the page-issues-X_sample hyphen notation, the EL Refine process normalizes them to all underscores before writing them to Hive. So the notation that the EL sanitization whitelist needs is all underscores. I will modify your patch and merge it (except for that detail, it LGTM!). I also will sanitize this schema retroactively for a couple months, since I'm already doing some maintenance work on EL sanitization pipeline. Hope this helps. Will ping you in this task when finished. Cheers!

Great, thanks a lot! The sample fields were introduced in September, so no need to go further back. (CC @Groceryheist )

@Tbayer event_sanitized.readingdepth is backfilled using the new whitelist.
I have vetted the resulting data and it looks good to me, but please do a quick check.
Note that the 28th of this month (in one week) we'll execute the purging script to delete data older than 90 days on the event database, and backfillings like this will not be possible any more (after 90 days).
Cheers!

In T216096#4976316, @mforns wrote:

@Tbayer event_sanitized.readingdepth is backfilled using the new whitelist.
I have vetted the resulting data and it looks good to me, but please do a quick check.

I spot-checked by comparing the daily number of events with each sample field set between the sanitized and unsanitized version, and they matched. Thanks again!

@Tbayer thanks for the check!