Page MenuHomePhabricator
Create Task
Maniphest T326596

Productionize db218[567]
Closed, ResolvedPublic
Actions

Description

These new hosts will replace db2093, db2094 and db2095

  • db2185 (replacing db2093 zarcillo)
  • db2186 (replacing db2094 - sanitarium)
  • db2187 (replacing db2095 - sanitarium)

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT326342 Q3:rack/setup/install db218[567]
 ResolvedMarosteguiT326584 Decommission db209[345]
 ResolvedRequestJhancock.wmT330975 decommission db2095.codfw.wmnet
 ResolvedMarosteguiT326596 Productionize db218[567]
 ResolvedRequestJhancock.wmT330827 decommission db2093.codfw.wmnet
 ResolvedRequestJhancock.wmT330828 decommission db2094.codfw.wmnet

Event Timeline

Marostegui created this task.Jan 9 2023, 10:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 9 2023, 10:30 PM
Marostegui triaged this task as Medium priority.Jan 9 2023, 10:30 PM
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T326584: Decommission db209[345].
Marostegui updated the task description. (Show Details)Jan 10 2023, 7:36 PM
Marostegui added a comment.Feb 27 2023, 10:41 AM

Extended their lv to the max after the installation.

Marostegui moved this task from Blocked to In progress on the DBA board.Feb 27 2023, 10:41 AM
Stashbot added a comment.Feb 28 2023, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2023-02-28T07:04:02Z] <marostegui> Stop mysql on db2094 T326596

gerritbot added a comment.Feb 28 2023, 7:07 AM

Change 892827 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2186

https://gerrit.wikimedia.org/r/892827

gerritbot added a project: Patch-For-Review.Feb 28 2023, 7:07 AM
gerritbot added a comment.Feb 28 2023, 8:14 AM

Change 892827 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2186

https://gerrit.wikimedia.org/r/892827

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2023, 8:30 AM
Marostegui added a comment.Feb 28 2023, 8:37 AM

Added db2186 to zarcillo

gerritbot added a comment.Feb 28 2023, 12:13 PM

Change 892939 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Do not reimage db2186

https://gerrit.wikimedia.org/r/892939

gerritbot added a project: Patch-For-Review.Feb 28 2023, 12:13 PM

Change 892939 merged by Marostegui:

[operations/puppet@production] install_server: Do not reimage db2186

https://gerrit.wikimedia.org/r/892939

gerritbot added a comment.Feb 28 2023, 12:24 PM

Change 892941 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2185

https://gerrit.wikimedia.org/r/892941

gerritbot added a comment.Feb 28 2023, 12:30 PM

Change 892941 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2185

https://gerrit.wikimedia.org/r/892941

gerritbot added a comment.Feb 28 2023, 12:42 PM

Change 892948 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbprov2002.cnf.erb: Change db_inventory target

https://gerrit.wikimedia.org/r/892948

Marostegui added a comment.Feb 28 2023, 12:44 PM

db2185 is now up and replicating. To consider it fully production we need to merge and test https://gerrit.wikimedia.org/r/c/operations/puppet/+/892948

gerritbot added a comment.Feb 28 2023, 12:58 PM

Change 892948 merged by Marostegui:

[operations/puppet@production] dbprov2002.cnf.erb: Change db_inventory target

https://gerrit.wikimedia.org/r/892948

gerritbot added a comment.Feb 28 2023, 1:01 PM

Change 892953 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] common.yaml: Add db2185

https://gerrit.wikimedia.org/r/892953

• jcrespo subscribed.Feb 28 2023, 1:01 PM

backup.png (731×1 px, 75 KB)

Marostegui added a comment.Feb 28 2023, 1:02 PM
In T326596#8652464, @jcrespo wrote:

backup.png (731×1 px, 75 KB)

Thanks!

gerritbot added a comment.Feb 28 2023, 1:04 PM

Change 892953 merged by Marostegui:

[operations/puppet@production] common.yaml: Add db2185

https://gerrit.wikimedia.org/r/892953

Marostegui updated the task description. (Show Details)Feb 28 2023, 1:09 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 28 2023, 1:12 PM
Marostegui added a comment.EditedFeb 28 2023, 1:57 PM

db2186 is now up and catching up. I am going to test all the scripts (like check data, redact sanitarium etc)

Stashbot added a comment.Feb 28 2023, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-28T13:59:42Z] <marostegui> Create dummy and empty enwiki.text table on db2186:3311 to test check_private_data T326596

Marostegui added a comment.Feb 28 2023, 2:00 PM

I have created a dummy enwiki.text table on db2186:3311 and it gets correctly seen by check_private_data.
Going to leave it there to make sure it is also caught by the cronjob

root@db2186:/srv#  check_private_data.py -S /run/mysqld/mysqld.s1.sock
-- Non-public databases that are present:
-- Non-public tables that are present:
DROP TABLE IF EXISTS `enwiki`.`text`;
-- Unfiltered columns that are present:
Marostegui added a comment.Mar 1 2023, 5:36 AM

The email warning about private data on db2186:3311 arrived correctly. I just removed the dummy table

Stashbot added a comment.Mar 1 2023, 5:37 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-01T05:37:21Z] <marostegui> Stop mysql on codfw sanitarium host db2095 (s2, s7, s6, s4) to clone db2187 T326596

gerritbot added a comment.Mar 1 2023, 6:11 AM

Change 893104 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Do not reimage db2185

https://gerrit.wikimedia.org/r/893104

gerritbot added a project: Patch-For-Review.Mar 1 2023, 6:11 AM
Marostegui added a subtask: T330827: decommission db2093.codfw.wmnet.Mar 1 2023, 6:11 AM
Marostegui added a subtask: T330828: decommission db2094.codfw.wmnet.
gerritbot added a comment.Mar 1 2023, 6:13 AM

Change 893104 merged by Marostegui:

[operations/puppet@production] install_server: Do not reimage db2185

https://gerrit.wikimedia.org/r/893104

Marostegui updated the task description. (Show Details)Mar 1 2023, 6:21 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 1 2023, 6:31 AM
gerritbot added a comment.Mar 1 2023, 7:47 AM

Change 893390 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db2187

https://gerrit.wikimedia.org/r/893390

gerritbot added a project: Patch-For-Review.Mar 1 2023, 7:47 AM

Change 893390 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db2187

https://gerrit.wikimedia.org/r/893390

gerritbot added a comment.Mar 1 2023, 7:49 AM

Change 893391 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Remove db2187 as insetup

https://gerrit.wikimedia.org/r/893391

gerritbot added a comment.Mar 1 2023, 7:50 AM

Change 893391 merged by Marostegui:

[operations/puppet@production] site.pp: Remove db2187 as insetup

https://gerrit.wikimedia.org/r/893391

Maintenance_bot removed a project: Patch-For-Review.Mar 1 2023, 8:10 AM
gerritbot added a comment.Mar 1 2023, 11:27 AM

Change 893420 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Add db2187

https://gerrit.wikimedia.org/r/893420

gerritbot added a project: Patch-For-Review.Mar 1 2023, 11:27 AM

Change 893420 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Add db2187

https://gerrit.wikimedia.org/r/893420

Maintenance_bot removed a project: Patch-For-Review.Mar 1 2023, 11:31 AM
Marostegui added a comment.Mar 1 2023, 11:57 AM

db2187 is now up and replicating s2, s4, s6 and s7.
I am checking the data to make sure triggers are working fine. I will also leave a dummy table on frwiki.text to see if it is reported correctly tomorrow morning by the automatic check - similar to what was done at T326596#8652681

root@db2187:/srv# check_private_data.py -S /run/mysqld/mysqld.s6.sock
-- Non-public databases that are present:
-- Non-public tables that are present:
DROP TABLE IF EXISTS `frwiki`.`text`;
-- Unfiltered columns that are present:
Marostegui added a comment.Mar 2 2023, 7:15 AM

The email warning about private data on db2187:3316 arrived correctly. I just removed the dummy table

Marostegui added parent tasks: T330975: decommission db2095.codfw.wmnet, T326584: Decommission db209[345].Mar 2 2023, 7:19 AM
gerritbot added a comment.Mar 2 2023, 7:29 AM

Change 893588 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Remove db2095

https://gerrit.wikimedia.org/r/893588

gerritbot added a project: Patch-For-Review.Mar 2 2023, 7:29 AM

Change 893588 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Remove db2095

https://gerrit.wikimedia.org/r/893588

Maintenance_bot removed a project: Patch-For-Review.Mar 2 2023, 7:31 AM
Marostegui closed this task as Resolved.Mar 2 2023, 8:59 AM
Marostegui updated the task description. (Show Details)

All green in Icinga, resolving.

Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 2 2023, 9:15 AM
gerritbot added a comment.Mar 3 2023, 5:55 AM

Change 893845 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Do not reimage db2187

https://gerrit.wikimedia.org/r/893845

gerritbot added a project: Patch-For-Review.Mar 3 2023, 5:55 AM

Change 893845 merged by Marostegui:

[operations/puppet@production] install_server: Do not reimage db2187

https://gerrit.wikimedia.org/r/893845

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2023, 6:10 AM
Jhancock.wm closed subtask T330828: decommission db2094.codfw.wmnet as Resolved.Mar 7 2023, 11:28 PM
Jhancock.wm closed subtask T330827: decommission db2093.codfw.wmnet as Resolved.Mar 9 2023, 3:04 PM
gerritbot added a comment.Mar 28 2023, 8:44 AM

Change 903602 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] orchestrator.conf.json.erb: Replace sanitarium hosts

https://gerrit.wikimedia.org/r/903602

gerritbot added a project: Patch-For-Review.Mar 28 2023, 8:44 AM
gerritbot added a comment.Mar 28 2023, 8:48 AM

Change 903602 merged by Marostegui:

[operations/puppet@production] orchestrator.conf.json.erb: Replace sanitarium hosts

https://gerrit.wikimedia.org/r/903602

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits