Page MenuHomePhabricator
Create Task
Maniphest T244312

Homepage: purge sanitized event data through 2019-11-04
Closed, ResolvedPublic
Actions

Description

In order to stay compliant with the Growth Team's data retention approvals for experiments, we would like to have some of the sanitized data related to the Homepage experiment deleted ASAP (rather than wait for the 270 day automatic deletion to roll around).

Per this diff to the Data Retention Guidelines, our rolling approval took effect on 2019-11-05. We are therefore asking to have data prior to, and including, 2019-11-04 deleted. The following tables are affected:

  • event_sanitized.homepagevisit
  • event_sanitized.homepagemodule
  • event_sanitized.helppanel

Please let @nettrom_WMF or @MMiller_WMF know what questions there might be about this.

Related Objects
Search...

Event Timeline

nettrom_WMF created this task.Feb 5 2020, 12:13 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 5 2020, 12:13 AM
nettrom_WMF mentioned this in T243557: Homepage: purge old data.Feb 5 2020, 12:15 AM
• fdans subscribed.Feb 6 2020, 6:09 PM

Do we need to delete all data in the tables or just some specific partitions?

• fdans claimed this task.Feb 6 2020, 6:10 PM
• fdans moved this task from Incoming to Ops Week on the Analytics board.
nettrom_WMF added a comment.Feb 6 2020, 6:44 PM
In T244312#5857137, @fdans wrote:

Do we need to delete all data in the tables or just some specific partitions?

Since these are EventLogging tables, they're partitioned by year/month/day/hour. So all partitions for dates <= 2019-11-04 should be deleted. (In other words, data from 2019-11-05 onwards should still be available)

nettrom_WMF edited projects, added Product-Analytics; removed Product-Analytics (Kanban).Feb 6 2020, 9:08 PM
nettrom_WMF moved this task from Triage to Tracking on the Product-Analytics board.
• fdans added a comment.Feb 18 2020, 2:24 PM

Purge complete!

For event_sanitized.homepagevisit

PYTHONPATH=/srv/deployment/analytics/refinery/python /srv/deployment/analytics/refinery/bin/refinery-drop-older-than -d event_sanitized -t event_sanitized.homepagevisit -b /wmf/data/event_sanitized/HomepageVisit --path-format='year=(?P<year>[0-9]+)(/month=(?P<month>[0-9]+)(/day=(?P<day>[0-9]+)(/hour=(?P<hour>[0-9]+))?)?)?' -o 2019-11-05

For event_sanitized.homepagemodule 

PYTHONPATH=/srv/deployment/analytics/refinery/python /srv/deployment/analytics/refinery/bin/refinery-drop-older-than -d event_sanitized -t event_sanitized.homepagemodule -b /wmf/data/event_sanitized/HomepageModule --path-format='year=(?P<year>[0-9]+)(/month=(?P<month>[0-9]+)(/day=(?P<day>[0-9]+)(/hour=(?P<hour>[0-9]+))?)?)?' -o 2019-11-05

For event_sanitized.helppanel 

PYTHONPATH=/srv/deployment/analytics/refinery/python /srv/deployment/analytics/refinery/bin/refinery-drop-older-than -d event_sanitized -t event_sanitized.helppanel -b /wmf/data/event_sanitized/HelpPanel --path-format='year=(?P<year>[0-9]+)(/month=(?P<month>[0-9]+)(/day=(?P<day>[0-9]+)(/hour=(?P<hour>[0-9]+))?)?)?' -o 2019-11-05
• fdans added a project: Analytics-Kanban.Feb 18 2020, 2:30 PM
• fdans moved this task from Next Up to Done on the Analytics-Kanban board.
nettrom_WMF closed this task as Resolved.Feb 18 2020, 3:01 PM

Verified in Hive that data is not available for any of the three schemas up through 2019-11-04 and this task can be closed. Thanks for your work on this @fdans !

nettrom_WMF mentioned this in T249666: Growth: validate that data is purged after 270 days.Aug 10 2020, 6:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL