Page MenuHomePhabricator

Rerun sanitization before archiving eventlogging mysql data
Closed, ResolvedPublic5 Story Points

Event Timeline

elukey added a comment.EditedOct 31 2019, 6:53 AM

As part of this task I'd also clean up profile::mariadb::misc::eventlogging::sanitization from the db110[78] hosts :)

Script to run:

elukey@db1107:~$ sudo systemctl cat eventlogging_db_sanitization | grep ExecStart
ExecStart=/usr/local/bin/eventlogging_cleaner --whitelist /srv/deployment/analytics/refinery/static_data/eventlogging/whitelist.yaml --yaml --older-than 90 --start-ts-file /srv/eventlogging_cleaner/eventlogging_cleaner --batch-size 10000 --sleep-between-batches 2

elukey@db1108:~$ sudo systemctl cat eventlogging_db_sanitization | grep ExecStart
ExecStart=/usr/local/bin/eventlogging_cleaner --whitelist /srv/deployment/analytics/refinery/static_data/eventlogging/whitelist.yaml --yaml --older-than 90 --start-ts-file /srv/eventlogging_cleaner/eventlogging_cleaner --batch-size 10000 --sleep-between-batches 2

I'd also stop the timer/service beforehand to avoid issues.

Ottomata triaged this task as High priority.Oct 31 2019, 4:39 PM
Ottomata moved this task from Incoming to Operational Excellence on the Analytics board.

Change 548142 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] eventlogging: allow sanitization script to run on all db records

https://gerrit.wikimedia.org/r/548142

elukey added a subscriber: mforns.Nov 4 2019, 9:02 AM

High level plan that I have in mind:

  • review/merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/548142/
  • run sudo -u eventlogcleaner /usr/local/bin/eventlogging_cleaner --whitelist /srv/deployment/analytics/refinery/static_data/eventlogging/whitelist.yaml --yaml --older-than 0 --start-ts-file /srv/eventlogging_cleaner/eventlogging_cleaner --batch-size 10000 --sleep-between-batches 2

@mforns to triple check :)

Change 548142 merged by Elukey:
[operations/puppet@production] eventlogging: allow sanitization script to run on all db records

https://gerrit.wikimedia.org/r/548142

Change 548318 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] eventlogging: run sanitization script on all the db records

https://gerrit.wikimedia.org/r/548318

Change 548318 merged by Elukey:
[operations/puppet@production] eventlogging: run sanitization script on all the db records

https://gerrit.wikimedia.org/r/548318

elukey added a comment.Nov 5 2019, 7:52 AM

Remaining steps:

  • wait for the timer to complete to have full sanitization
  • clean up sanitization stuff from db nodes
  • clean up remaining puppet leftovers
elukey claimed this task.Nov 5 2019, 4:56 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

Sanitization is still running on the two databases!

Change 549070 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::mariadb::misc::eventlogging::sanitization: ease clean up

https://gerrit.wikimedia.org/r/549070

Change 549071 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove Eventloggging sanitization automation from log databases

https://gerrit.wikimedia.org/r/549071

Change 549070 merged by Elukey:
[operations/puppet@production] profile::mariadb::misc::eventlogging::sanitization: ease clean up

https://gerrit.wikimedia.org/r/549070

Change 549071 merged by Elukey:
[operations/puppet@production] Remove Eventloggging sanitization automation from log databases

https://gerrit.wikimedia.org/r/549071

elukey set the point value for this task to 5.Nov 12 2019, 11:42 AM
elukey moved this task from In Progress to Done on the Analytics-Kanban board.

Change 550738 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/scap@master] Remove db1107 and db1108 from scap targets

https://gerrit.wikimedia.org/r/550738

Change 550738 merged by Ottomata:
[analytics/refinery/scap@master] Remove db1107 and db1108 from scap targets

https://gerrit.wikimedia.org/r/550738

Nuria closed this task as Resolved.Thu, Nov 14, 7:46 PM