Page MenuHomePhabricator

[EL sanitization] Modify mysql purging script to read from the new YAML whitelist
Closed, ResolvedPublic8 Estimate Story Points

Description

The whitelist will be translated to YAML format, so the purging script will have to adapt to the new format.
Note that the new YAML whitelist allows for partial purging of nested fields!
The mysql purging script can not handle this kind of sanitizations, hence it will have to ignore nested fields that need partial purge and treat them as if they were completely whitelisted, and keep them in their entirety.

Details

Event Timeline

mforns created this task.Mar 14 2018, 3:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2018, 3:46 PM
mforns claimed this task.Mar 16 2018, 4:03 PM
mforns set the point value for this task to 8.
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 420685 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] Modify eventlogging purging script to read from YAML whitelist

https://gerrit.wikimedia.org/r/420685

mforns added a comment.EditedApr 4 2018, 2:32 PM

Still TODO:

  • Review and merge https://gerrit.wikimedia.org/r/#/c/420685/
  • Create a puppet change that replaces the whitelist AND adds the --yaml true flag to the cleaner script call
  • Check purging after first execution with the new whitelist

Change 420685 merged by Elukey:
[operations/puppet@production] Modify eventlogging purging script to read from YAML whitelist

https://gerrit.wikimedia.org/r/420685

Change 425496 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::el::sanitization: add package

https://gerrit.wikimedia.org/r/425496

Change 425496 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::el::sanitization: add package

https://gerrit.wikimedia.org/r/425496

Change 425498 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role:mariadb::misc::el::replica: add new yaml whitelist to db1108

https://gerrit.wikimedia.org/r/425498

Change 425498 merged by Elukey:
[operations/puppet@production] role:mariadb::misc::el::replica: add new yaml whitelist to db1108

https://gerrit.wikimedia.org/r/425498

elukey moved this task from In Code Review to Done on the Analytics-Kanban board.Apr 13 2018, 7:47 AM
elukey added a subscriber: elukey.Apr 13 2018, 7:50 AM
elukey@db1108:~$ sudo -u eventlogcleaner crontab -l
0 11 * * * /usr/bin/flock --verbose -n /var/lock/eventlogging_cleaner /usr/local/bin/eventlogging_cleaner --whitelist /etc/analytics/sanitization/eventlogging_purging_whitelist.yaml --yaml --older-than 90 --start-ts-file /var/run/eventlogging_cleaner --batch-size 10000 --sleep-between-batches 2  >> /var/log/eventlogging_cleaner/eventlogging_cleaner.log
elukey@db1107:~$ sudo -u eventlogcleaner crontab -l
0 11 * * * /usr/bin/flock --verbose -n /var/lock/eventlogging_cleaner /usr/local/bin/eventlogging_cleaner --whitelist /etc/analytics/sanitization/eventlogging_purging_whitelist.yaml --yaml --older-than 90 --start-ts-file /var/run/eventlogging_cleaner --batch-size 10000 --sleep-between-batches 2  >> /var/log/eventlogging_cleaner/eventlogging_cleaner.log

No more tsv in puppet and on the host, all yaml from now on :)

Nuria closed this task as Resolved.Apr 17 2018, 3:01 AM