Hi @Nuria. Please associate at least one project with this task, otherwise nobody can find this task when searching in the corresponding project(s). Thanks! :)

Question: Does this involve removing values of country data and/or access_method as well, or just the more fine-grained user_agent_map and city? (It's not clear to from the linked Phabricator task and mediawiki.org page.) Some people were interested earlier in analyses that would need country data from June.

For now we are concerned just with user agent map.

Task breakdown:

1. Write an oozie coordinator to backfill sanitization from exisiting pageview_hourly to pageview_hourly_new (path TBD)
2. Backfill by hand: create pageview_hourly_new, launch and monitor oozie job from 1.
3. Modify existing pageview_hourly oozie to include the HQL from 1 in the workflow --> Insert data from webrequest into pageview_hourly_tmp, then sanitize it into pageview_hourly.
4. When backfill is done, deploy
   • Stop pageview_hourly job, and wait for jobs using it to be waiting on new data.
   • Drop pageview_hourly and pageview_hourly_new tables in hiven
   • Rename data folders: pageview_hourly becomes pageview_hourly_tmp, and pageview_hourly_new becomes pageview_hourly.
   • Recreate pageview_hourly and pageview_hourly_tmp tables using new path from previous step.
   • Recreate partitions for existing data for pageview_hourly and pageview_hourly_tmp
   • Deploy the oozie change from 3 and run it starting at backfilling stop time.

Hopefully that should do it :)

Rework this to be more generic and allow a differentially private approach instead of k-anonimizing.