Key decisions from today's meeting with @JAllemandou and @lbowmaker

• changing the request to add one new column referer_name, since referer_class has already been updated with external (media sites) (updated the task description)
• adding the referer_name has PII implications. We will need to reach out to Legal, Safety and Security Service Center to get sign off before adding this field
• the additional field will not have cardinality issues in Druid as the number of values in the referer_name are < 100
• Note for Data Engineer : in order to ensure Hive and Spark are aligned, it is advisable to do these updates on SparkSQL (which will automatically update the Hive metastore).

Update: Privacy request submitted on 3/7. Responded for additional information on 3/14.

We have gotten the okay from Privacy team. See Privacy review - Pageview referrer column 20230314
"Overall, the proposed change already embeds adequate privacy safeguard. Therefore, it poses a low level of privacy risk, which can be automatically accepted by Product Analytics, as per the Security team’s risk management policy."

And we have approval from Legal as well ! L3SC Privacy Request: Product Analytics - Track referer name

Privacy legal conclusion: low risk

Public-facing: While our internal tables collect pageviews based on views per hour and views per day (and some of these external sites do have less than 1000 views per hour, for instance: Coursera), we only publish metrics for monthly pageviews. When considering the monthly pageviews, even the views for the smaller sites on the list consistently add up to more than 1000 views. This attends to our recommendations on only publishing data referring to subsets of >1000 people in order to keep it low-risk. Referrer names for individual articles, user pages and commons page will not be published.

Internal-facing: Although the internal metrics may store data on less visited sites, the fact that we will only be collecting the referrer name for 14 media sites (Youtube, Twitter, Facebook, Reddit, TikTok, Quora, Instagram, ycombinator, LinkedIn, Github, Pinterest, Coursera, Medium, Stackoverflow) and 23 search engines (see list here) attends to data minimization concerns and alleviates the re-identification risk. Users from other media sites/search engines outside of this list will not have a value for referer_name.The re-identification risk was also deemed low by Privacy Engineering.

@lbowmaker / @JArguello-WMF, now that we have approval, can we pls prioritize this task for the next DE sprint?
cc @JAllemandou

We can populate the Druid pageview tables as part of https://phabricator.wikimedia.org/T334104, no?