Page MenuHomePhabricator
Create Task
Maniphest T152188

Restart pending mysql hosts with old TLS cert
Closed, ResolvedPublic
Actions

Description

db1063.eqiad.wmnet
db1054.eqiad.wmnet
db1067.eqiad.wmnet
db1036.eqiad.wmnet
db1015.eqiad.wmnet
db1021.eqiad.wmnet
db1022.eqiad.wmnet

db2059.codfw.wmnet
db2035.codfw.wmnet
db2051.codfw.wmnet
db2061.codfw.wmnet
db2044.codfw.wmnet
db2052.codfw.wmnet
db2058.codfw.wmnet
db2066.codfw.wmnet
db2064.codfw.wmnet
db2065.codfw.wmnet
db2053.codfw.wmnet
db2041.codfw.wmnet
db2050.codfw.wmnet
db2045.codfw.wmnet
db2037.codfw.wmnet
db2046.codfw.wmnet
db2063.codfw.wmnet
db2036.codfw.wmnet
db2067.codfw.wmnet
db2060.codfw.wmnet
db2043.codfw.wmnet
db2054.codfw.wmnet
db2039.codfw.wmnet

Details

SubjectRepoBranchLines +/-
mariadb: depool db1015 for maintenanceoperations/mediawiki-configmaster+7 -7
mariadb: Depool db1022 for maintenanceoperations/mediawiki-configmaster+3 -3
mariadb: Depool db1021 for a quick rebootoperations/mediawiki-configmaster+4 -4
mariadb: Depool db1036 for a quick rebootoperations/mediawiki-configmaster+7 -7
eventlogging-mariadb: Add new TLS certs to eventlogging seversoperations/puppetproduction+19 -7
analytics-mariadb: Enable new certificates on eventlogging serversoperations/puppetproduction+29 -0
backups: Fix & uniform predump and bpipe mysql method of backupsoperations/puppetproduction+28 -10
Renew expired TLS certificate for eventlogging hostsoperations/puppetproduction+1 -1
mariadb: Update dbstores to use the latest TLS certificateoperations/puppetproduction+13 -1
Enable new TLS certs on labsdb hostsoperations/puppetproduction+2 -1
Repool db1060 after maintenanceoperations/mediawiki-configmaster+2 -2
Really depool db1060, unlike 2 patches agooperations/mediawiki-configmaster+1 -1
Pool db1074 with full load after warmupoperations/mediawiki-configmaster+1 -1
Depool db1060 for maintenanceoperations/mediawiki-configmaster+1 -1
Repool db1074 with low load after maintenanceoperations/mediawiki-configmaster+1 -1
Repool db1076 with full loadoperations/mediawiki-configmaster+1 -1
Depool db1074 for maintenance and upgradeoperations/mediawiki-configmaster+1 -1
Repool db1076 with low load after maintenanceoperations/mediawiki-configmaster+1 -1
Depool db1076 for maintenanceoperations/mediawiki-configmaster+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• jcrespo created this task.Dec 2 2016, 8:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 2 2016, 8:56 AM
gerritbot subscribed.Dec 2 2016, 8:57 AM

Change 324862 had a related patch set uploaded (by Jcrespo):
Depool db1076 for maintenance

https://gerrit.wikimedia.org/r/324862

gerritbot added a project: Patch-For-Review.Dec 2 2016, 8:57 AM
gerritbot added a comment.Dec 2 2016, 9:00 AM

Change 324862 merged by Jcrespo:
Depool db1076 for maintenance

https://gerrit.wikimedia.org/r/324862

Stashbot subscribed.Dec 2 2016, 9:18 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-02T09:18:14Z] <jynus> mysql restart and upgrade for db1076 T152188

gerritbot added a comment.Dec 2 2016, 9:24 AM

Change 324867 had a related patch set uploaded (by Jcrespo):
Repool db1076 with low load after maintenance

https://gerrit.wikimedia.org/r/324867

gerritbot added a comment.Dec 2 2016, 9:51 AM

Change 324867 merged by Jcrespo:
Repool db1076 with low load after maintenance

https://gerrit.wikimedia.org/r/324867

gerritbot added a comment.Dec 2 2016, 9:56 AM

Change 324869 had a related patch set uploaded (by Jcrespo):
Depool db1074 for maintenance and upgrade

https://gerrit.wikimedia.org/r/324869

gerritbot added a comment.Dec 2 2016, 10:07 AM

Change 324869 merged by Jcrespo:
Depool db1074 for maintenance and upgrade

https://gerrit.wikimedia.org/r/324869

gerritbot added a comment.Dec 2 2016, 10:26 AM

Change 324874 had a related patch set uploaded (by Jcrespo):
Repool db1076 with full load

https://gerrit.wikimedia.org/r/324874

Stashbot added a comment.Dec 2 2016, 10:28 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-02T10:28:44Z] <jynus> mysql restart and upgrade for db1074 T152188

gerritbot added a comment.Dec 2 2016, 10:31 AM

Change 324874 merged by jenkins-bot:
Repool db1076 with full load

https://gerrit.wikimedia.org/r/324874

gerritbot added a comment.Dec 2 2016, 10:46 AM

Change 324879 had a related patch set uploaded (by Jcrespo):
Repool db1074 with low load after maintenance

https://gerrit.wikimedia.org/r/324879

gerritbot added a comment.Dec 2 2016, 10:58 AM

Change 324879 merged by jenkins-bot:
Repool db1074 with low load after maintenance

https://gerrit.wikimedia.org/r/324879

gerritbot added a comment.Dec 2 2016, 11:36 AM

Change 324885 had a related patch set uploaded (by Jcrespo):
Depool db1060 for maintenance

https://gerrit.wikimedia.org/r/324885

gerritbot added a comment.Dec 2 2016, 11:38 AM

Change 324885 merged by Jcrespo:
Depool db1060 for maintenance

https://gerrit.wikimedia.org/r/324885

gerritbot added a comment.Dec 2 2016, 11:43 AM

Change 324886 had a related patch set uploaded (by Jcrespo):
Pool db1074 with full load after warmup

https://gerrit.wikimedia.org/r/324886

gerritbot added a comment.Dec 2 2016, 11:48 AM

Change 324888 had a related patch set uploaded (by Jcrespo):
Really depool db1060, unlike 2 patches ago

https://gerrit.wikimedia.org/r/324888

gerritbot added a comment.Dec 2 2016, 11:49 AM

Change 324886 merged by jenkins-bot:
Pool db1074 with full load after warmup

https://gerrit.wikimedia.org/r/324886

gerritbot added a comment.Dec 2 2016, 11:49 AM

Change 324888 merged by jenkins-bot:
Really depool db1060, unlike 2 patches ago

https://gerrit.wikimedia.org/r/324888

• jcrespo claimed this task.Dec 2 2016, 11:54 AM
• jcrespo moved this task from Triage to In progress on the DBA board.
Stashbot added a comment.Dec 2 2016, 11:56 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-02T11:56:38Z] <jynus> mysql restart for db1060 T152188

gerritbot added a comment.Dec 2 2016, 12:25 PM

Change 324894 had a related patch set uploaded (by Jcrespo):
Repool db1060 after maintenance

https://gerrit.wikimedia.org/r/324894

gerritbot added a comment.Dec 2 2016, 1:07 PM

Change 324894 merged by jenkins-bot:
Repool db1060 after maintenance

https://gerrit.wikimedia.org/r/324894

gerritbot added a comment.Dec 2 2016, 1:16 PM

Change 324899 had a related patch set uploaded (by Jcrespo):
Enable new TLS certs on labsdb hosts

https://gerrit.wikimedia.org/r/324899

gerritbot added a comment.Dec 2 2016, 1:25 PM

Change 324899 merged by Jcrespo:
Enable new TLS certs on labsdb hosts

https://gerrit.wikimedia.org/r/324899

• jcrespo lowered the priority of this task from Medium to Low.Dec 2 2016, 2:06 PM
• jcrespo moved this task from In progress to Backlog on the DBA board.
gerritbot added a comment.Dec 2 2016, 2:11 PM

Change 324908 had a related patch set uploaded (by Jcrespo):
mariadb: Update dbstores to use the latest TLS certificate

https://gerrit.wikimedia.org/r/324908

gerritbot added a comment.Dec 2 2016, 2:36 PM

Change 324908 merged by Jcrespo:
mariadb: Update dbstores to use the latest TLS certificate

https://gerrit.wikimedia.org/r/324908

• jcrespo mentioned this in T152364: db1047 out of disk space, eventlogging_sync spam.Dec 5 2016, 10:29 AM
• jcrespo added a subtask: T152364: db1047 out of disk space, eventlogging_sync spam.
gerritbot added a comment.Dec 5 2016, 10:36 AM

Change 325273 had a related patch set uploaded (by Jcrespo):
Renew expired TLS certificate for eventlogging hosts

https://gerrit.wikimedia.org/r/325273

• jcrespo closed subtask T152364: db1047 out of disk space, eventlogging_sync spam as Resolved.Dec 7 2016, 9:20 AM
gerritbot added a comment.Dec 7 2016, 9:44 AM

Change 325273 merged by Jcrespo:
Renew expired TLS certificate for eventlogging hosts

https://gerrit.wikimedia.org/r/325273

Stashbot added a comment.Dec 7 2016, 10:03 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-07T10:03:42Z] <jynus> restart and upgrade of dbstore1001 T152188

gerritbot added a comment.Dec 7 2016, 11:17 AM

Change 325759 had a related patch set uploaded (by Jcrespo):
Fixes to the predump and bpipe mysql method of backups

https://gerrit.wikimedia.org/r/325759

Stashbot added a comment.Dec 7 2016, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-07T14:19:05Z] <jynus> restart and upgrade of dbstore200[12] T152188

gerritbot added a comment.Dec 7 2016, 5:28 PM

Change 325759 merged by Jcrespo:
backups: Fix & uniform predump and bpipe mysql method of backups

https://gerrit.wikimedia.org/r/325759

• jcrespo added a comment.Dec 7 2016, 6:08 PM

We finally have the backups again up and running, with one day of delay. Reminder: check that all complete ok.

Stashbot added a comment.Dec 9 2016, 1:47 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-09T13:47:27Z] <jynus> disable puppet on db1047, db1046 and dbstore1002 in preparation for restarts T152188

Stashbot added a comment.Dec 9 2016, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-09T14:06:17Z] <jynus> restarting db1046 T152188

gerritbot added a comment.Dec 9 2016, 2:11 PM

Change 326122 had a related patch set uploaded (by Jcrespo):
analytics-mariadb: Enable new certificates on eventlogging servers

https://gerrit.wikimedia.org/r/326122

gerritbot added a comment.Dec 9 2016, 2:13 PM

Change 326122 merged by Jcrespo:
analytics-mariadb: Enable new certificates on eventlogging servers

https://gerrit.wikimedia.org/r/326122

gerritbot added a comment.Dec 9 2016, 2:20 PM

Change 326123 had a related patch set uploaded (by Jcrespo):
eventlogging-mariadb: Add new TLS certs to eventlogging severs

https://gerrit.wikimedia.org/r/326123

gerritbot added a comment.Dec 9 2016, 2:21 PM

Change 326123 merged by Jcrespo:
eventlogging-mariadb: Add new TLS certs to eventlogging severs

https://gerrit.wikimedia.org/r/326123

Stashbot added a comment.Dec 9 2016, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-09T14:39:23Z] <jynus> restarting db1047 T152188

Stashbot added a comment.Dec 9 2016, 2:49 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-09T14:49:39Z] <jynus> restarting dbstore1002 T152188

• jcrespo added a comment.Dec 14 2016, 6:20 PM

Pending hosts:

db1063.eqiad.wmnet
db1054.eqiad.wmnet
db1067.eqiad.wmnet
db1036.eqiad.wmnet
db1015.eqiad.wmnet
db1021.eqiad.wmnet
db1022.eqiad.wmnet

db2059.codfw.wmnet
db2035.codfw.wmnet
db2051.codfw.wmnet
db2061.codfw.wmnet
db2044.codfw.wmnet
db2052.codfw.wmnet
db2058.codfw.wmnet
db2066.codfw.wmnet
db2064.codfw.wmnet
db2065.codfw.wmnet
db2053.codfw.wmnet
db2041.codfw.wmnet
db2050.codfw.wmnet
db2045.codfw.wmnet
db2037.codfw.wmnet
db2046.codfw.wmnet
db2063.codfw.wmnet
db2036.codfw.wmnet
db2067.codfw.wmnet
db2060.codfw.wmnet
db2043.codfw.wmnet
db2054.codfw.wmnet
db2039.codfw.wmnet
• jcrespo updated the task description. (Show Details)Dec 14 2016, 6:21 PM
• jcrespo added a comment.Feb 3 2017, 6:27 PM

Pending hosts:

db1036.eqiad.wmnet: cacert
db1021.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db2050.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert
Stashbot added a comment.Feb 6 2017, 6:16 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-06T18:16:21Z] <jynus> preparing to reimage db2050 T152188

Stashbot added a comment.Feb 6 2017, 7:13 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-06T19:13:58Z] <jynus> preparing to reimage db2045 T152188

ops-monitoring-bot subscribed.Feb 6 2017, 9:31 PM

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2050.codfw.wmnet', 'db2045.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702062131_jynus_29631.log.

ops-monitoring-bot added a comment.Feb 6 2017, 10:02 PM

Completed auto-reimage of hosts:

['db2050.codfw.wmnet', 'db2045.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Feb 7 2017, 10:34 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-07T10:34:38Z] <jynus> preparing db2046 for reimage T152188

gerritbot added a comment.Feb 7 2017, 10:51 AM

Change 336391 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1036 for a quick reboot

https://gerrit.wikimedia.org/r/336391

gerritbot added a comment.Feb 7 2017, 11:28 AM

Change 336391 merged by jenkins-bot:
mariadb: Depool db1036 for a quick reboot

https://gerrit.wikimedia.org/r/336391

ops-monitoring-bot added a comment.Feb 7 2017, 11:49 AM

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2046.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702071149_jynus_5780.log.

ops-monitoring-bot added a comment.Feb 7 2017, 12:17 PM

Completed auto-reimage of hosts:

['db2046.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Feb 7 2017, 12:35 PM

Change 336402 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1021 for a quick reboot

https://gerrit.wikimedia.org/r/336402

gerritbot added a comment.Feb 7 2017, 12:39 PM

Change 336402 merged by jenkins-bot:
mariadb: Depool db1021 for a quick reboot

https://gerrit.wikimedia.org/r/336402

Stashbot added a comment.Feb 7 2017, 2:56 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-07T14:56:24Z] <jynus> preparing db2036 for reimage T152188

Stashbot added a comment.Feb 7 2017, 4:08 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-07T16:08:17Z] <jynus> restarting and upgrading db2064 T152188

gerritbot added a comment.Feb 7 2017, 4:30 PM

Change 336429 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1022 for maintenance

https://gerrit.wikimedia.org/r/336429

gerritbot added a comment.Feb 7 2017, 4:35 PM

Change 336429 merged by Jcrespo:
mariadb: Depool db1022 for maintenance

https://gerrit.wikimedia.org/r/336429

Stashbot added a comment.Feb 7 2017, 5:07 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-07T17:07:47Z] <jynus> restarting and upgrading db1022 T152188

ops-monitoring-bot added a comment.Feb 7 2017, 5:25 PM

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2036.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702071725_jynus_1180.log.

ops-monitoring-bot added a comment.Feb 7 2017, 5:47 PM

Completed auto-reimage of hosts:

['db2036.codfw.wmnet']

Of which those FAILED:

set(['db2036.codfw.wmnet'])
gerritbot added a comment.Feb 7 2017, 6:33 PM

Change 336453 had a related patch set uploaded (by Jcrespo):
mariadb: depool db1015 for maintenance

https://gerrit.wikimedia.org/r/336453

gerritbot added a comment.Feb 7 2017, 6:53 PM

Change 336453 merged by jenkins-bot:
mariadb: depool db1015 for maintenance

https://gerrit.wikimedia.org/r/336453

Stashbot added a comment.Feb 7 2017, 6:55 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-07T18:55:55Z] <jynus> restarting and upgrading db1015 T152188

Stashbot added a comment.Feb 7 2017, 6:58 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-07T18:58:18Z] <jynus> preparing db2043 for reimage T152188

ops-monitoring-bot added a comment.Feb 7 2017, 9:35 PM

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2043.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702072134_jynus_5655.log.

ops-monitoring-bot added a comment.Feb 7 2017, 10:03 PM

Completed auto-reimage of hosts:

['db2043.codfw.wmnet']

and were ALL successful.

• jcrespo closed this task as Resolved.Feb 8 2017, 12:24 AM

All hosts with the old expiring cert have been reimagened or (if scheduled for decomission), restarted:

sudo salt --output=txt -C 'G@cluster:mysql' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep cacert

Old, non-puppet certs can be removed and puppetization changed. To be done at T111654.

• jcrespo mentioned this in T111654: Set up TLS for MariaDB replication.Feb 8 2017, 12:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits