The sanitization list has been cleaned from quite some schemas in https://gerrit.wikimedia.org/r/c/analytics/refinery/+/755724.
We should delete associated historical data if no one uses it.
Here is the list of events concerned:
Most event-types don't even have a folder, some have data:
joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/edit ls: `/wmf/data/event_sanitized/edit': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/diacritics* ls: `/wmf/data/event_sanitized/diacritics': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebbrowse ls: `/wmf/data/event_sanitized/mobilewebbrowse': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebclicktracking ls: `/wmf/data/event_sanitized/mobilewebclicktracking': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebcta ls: `/wmf/data/event_sanitized/mobilewebcta': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebdiffclicktracking ls: `/wmf/data/event_sanitized/mobilewebdiffclicktracking': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebediting ls: `/wmf/data/event_sanitized/mobilewebediting': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebinfobox ls: `/wmf/data/event_sanitized/mobilewebinfobox': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobileweblanguageswitcher Found 3 items drwxr-x--- - analytics analytics-privatedata-users 0 2018-06-21 18:06 /wmf/data/event_sanitized/mobileweblanguageswitcher/year=2017 drwxr-x--- - analytics analytics-privatedata-users 0 2018-08-11 03:07 /wmf/data/event_sanitized/mobileweblanguageswitcher/year=2018 drwxr-x--- - analytics analytics-privatedata-users 0 2019-10-22 18:04 /wmf/data/event_sanitized/mobileweblanguageswitcher/year=2019 joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebmainmenuclicktracking Found 4 items drwxr-x--- - analytics analytics-privatedata-users 0 2018-06-25 18:39 /wmf/data/event_sanitized/mobilewebmainmenuclicktracking/year=2017 drwxr-x--- - analytics analytics-privatedata-users 0 2018-12-01 04:40 /wmf/data/event_sanitized/mobilewebmainmenuclicktracking/year=2018 drwxr-x--- - analytics analytics-privatedata-users 0 2019-12-01 21:03 /wmf/data/event_sanitized/mobilewebmainmenuclicktracking/year=2019 drwxr-x--- - analytics analytics-privatedata-users 0 2020-02-01 15:04 /wmf/data/event_sanitized/mobilewebmainmenuclicktracking/year=2020 joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebsectionusage Found 3 items drwxr-x--- - analytics analytics-privatedata-users 0 2018-08-12 17:04 /wmf/data/event_sanitized/mobilewebsectionusage/year=2018 drwxr-x--- - analytics analytics-privatedata-users 0 2019-09-03 18:03 /wmf/data/event_sanitized/mobilewebsectionusage/year=2019 drwxr-x--- - analytics analytics-privatedata-users 0 2021-02-09 03:04 /wmf/data/event_sanitized/mobilewebsectionusage/year=2021 joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebsharebutton ls: `/wmf/data/event_sanitized/mobilewebsharebutton': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebuiclicktracking Found 4 items drwxr-x--- - analytics analytics-privatedata-users 0 2018-06-21 23:20 /wmf/data/event_sanitized/mobilewebuiclicktracking/year=2017 drwxr-x--- - analytics analytics-privatedata-users 0 2018-12-02 02:02 /wmf/data/event_sanitized/mobilewebuiclicktracking/year=2018 drwxr-x--- - analytics analytics-privatedata-users 0 2019-12-15 04:03 /wmf/data/event_sanitized/mobilewebuiclicktracking/year=2019 drwxr-x--- - analytics analytics-privatedata-users 0 2020-08-09 00:04 /wmf/data/event_sanitized/mobilewebuiclicktracking/year=2020 joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebuploads ls: `/wmf/data/event_sanitized/mobilewebuploads': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebwatching* ls: `/wmf/data/event_sanitized/mobilewebwatching*': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/mobilewebwikigrok* ls: `/wmf/data/event_sanitized/mobilewebwikigrok*': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/taskrecommendation* ls: `/wmf/data/event_sanitized/taskrecommendation*': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/testsearchsatisfaction ls: `/wmf/data/event_sanitized/testsearchsatisfaction': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/vet135171 ls: `/wmf/data/event_sanitized/vet135171': No such file or directory joal@stat1008:~$ hdfs dfs -ls /wmf/data/event_sanitized/wikipediazerousage Found 6 items drwxr-x--- - analytics analytics-privatedata-users 0 2018-06-25 19:56 /wmf/data/event_sanitized/wikipediazerousage/year=2017 drwxr-x--- - analytics analytics-privatedata-users 0 2018-12-06 04:01 /wmf/data/event_sanitized/wikipediazerousage/year=2018 drwxr-x--- - analytics analytics-privatedata-users 0 2019-12-13 21:03 /wmf/data/event_sanitized/wikipediazerousage/year=2019 drwxr-x--- - analytics analytics-privatedata-users 0 2020-07-10 03:04 /wmf/data/event_sanitized/wikipediazerousage/year=2020 drwxr-x--- - analytics analytics-privatedata-users 0 2021-12-10 05:04 /wmf/data/event_sanitized/wikipediazerousage/year=2021 drwxr-x--- - analytics analytics-privatedata-users 0 2022-09-28 19:04 /wmf/data/event_sanitized/wikipediazerousage/year=2022
In terms of files (output is in the form DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILENAME):
joal@stat1008:~$ hdfs dfs -count /wmf/data/event_sanitized/mobileweblanguageswitcher
22 18 42608 /wmf/data/event_sanitized/mobileweblanguageswitcher
joal@stat1008:~$ hdfs dfs -count /wmf/data/event_sanitized/mobilewebmainmenuclicktracking
15768 73912 3143463930 /wmf/data/event_sanitized/mobilewebmainmenuclicktracking
joal@stat1008:~$ hdfs dfs -count /wmf/data/event_sanitized/mobilewebsectionusage
28 22 46216 /wmf/data/event_sanitized/mobilewebsectionusage
joal@stat1008:~$ hdfs dfs -count /wmf/data/event_sanitized/mobilewebuiclicktracking
952 1909 3994600 /wmf/data/event_sanitized/mobilewebuiclicktracking
joal@stat1008:~$ hdfs dfs -count /wmf/data/event_sanitized/wikipediazerousage
6201 19039 47408689 /wmf/data/event_sanitized/wikipediazerousageMay I proceed with deleting the tables from the Hive metastore for the impacted datasets?
Data has been deleted from HDFS. It will be quarantined in hdfs://analytics-hadoop/user/hdfs/.Trash/Current/wmf/data/event_sanitized for a period longer than the on week grace time required by this task.
@JAllemandou could you ack if it's ok to move ahead and delete related tables from event?