Page MenuHomePhabricator

Create a sanitarium redaction cookbook
Closed, ResolvedPublic

Description

Once a new wiki is created we need to sanitize the data and create grants on sanitarium hosts. This process should ideally done via cookbook, the process would be something like:

cookbook $HOSTS $SECTION $WIKINAME -t $TASKID

Hosts: Sanitarium hosts
Section: The section where the new wiki was created (by default as of today is s5 but this could change)
Wikiname: The wiki to be sanitized
Phabricator task regarding to the wiki storage creation, example task: T314645

  • Locally: Run the following on each host: redact_sanitarium.sh -d $WIKI -S /run/mysqld/mysqld.$SECTION.sock | mysql -S /run/mysqld/mysqld.$SECTION.sock $WIKI
  • Locally: Run the following on each host: check_private_data.py -S /run/mysqld/mysqld.$SECTION.sock | mysql -S /run/mysqld/mysqld.$SECTION.sock
  • Locally: Double check all private data is gone by running: check_private_data.py -S /run/mysqld/mysqld.$SECTION.sock

If the above is successful run the following - if the data isn't sanitized or any of the above steps above it should abort and never continue with the following steps:

  • Remotely (can be done from cumin hosts by doing db-mysql $HOST:$PORT: Connect to the sanitarium host section and grab the clouddb hosts that hang from them by running: "show slave hosts"
  • Remotely connect to those clouddb hosts and run the following commands:
set session sql_log_bin=0;
create database $WIKI_p;
GRANT SELECT, SHOW VIEW ON `$WIKI_p`.* TO `labsdbuser`;

Once the above is done, if possible create an automated comment on the phabricator task saying:

  • Wiki sanitized
  • Database "_p" created
  • Grants assigned to labsdbuser
  • Wiki ready for views creation

Event Timeline

ABran-WMF triaged this task as Medium priority.
ABran-WMF moved this task from Triage to Ready on the DBA board.

thanks for the creation @Marostegui !

Change #1080129 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/cookbooks@master] mariadb: pii cleaner cookbook

https://gerrit.wikimedia.org/r/1080129

ABran-WMF removed a project: DBA.

cookbook has been tested T375016#10257795 and other newly created wikis that were alarming on PII
pending review to be merged

ABran-WMF changed the task status from Open to Stalled.Oct 24 2024, 7:37 AM

Change #1090859 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] aliases.yaml: add db-clouddb-sanitzation

https://gerrit.wikimedia.org/r/1090859

ABran-WMF changed the task status from Stalled to In Progress.Nov 13 2024, 2:23 PM

Change #1090859 merged by Arnaudb:

[operations/puppet@production] aliases.yaml: add db-clouddb-sanitzation

https://gerrit.wikimedia.org/r/1090859

Change #1080129 merged by Federico Ceratto:

[operations/cookbooks@master] sre.mysql.sanitize-wiki: sanitize wiki cookbook

https://gerrit.wikimedia.org/r/1080129

$ sudo cookbook sre.mysql.sanitize-wiki
usage: cookbook [GLOBAL_ARGS] sre.mysql.sanitize-wiki [-h] --wiki WIKI [--check-only] --task TASK [--only-grant-and-view]
cookbook [GLOBAL_ARGS] sre.mysql.sanitize-wiki: error: the following arguments are required: --wiki, --task

Nice :)
I think this task can be resolved until we test it, if there's something missing/wrong, we can reopen

Change #1139035 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/cookbooks@master] sre.mysql.sanitize-wiki - handle multiple hosts

https://gerrit.wikimedia.org/r/1139035

Change #1139035 merged by Federico Ceratto:

[operations/cookbooks@master] sre.mysql.sanitize-wiki - handle multiple hosts

https://gerrit.wikimedia.org/r/1139035

Change #1191689 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/cookbooks@master] sanitize-wiki.py: Improve logging

https://gerrit.wikimedia.org/r/1191689

Change #1191689 merged by Federico Ceratto:

[operations/cookbooks@master] sanitize-wiki.py: Improve logging

https://gerrit.wikimedia.org/r/1191689