Page MenuHomePhabricator

Rolling restart of external storage servers for TLS certificate update
Closed, ResolvedPublic

Details

Related Gerrit Patches:
operations/mediawiki-config : masterRepool es2019 after maintenance
operations/mediawiki-config : masterDepool es1015 for maintenance
operations/dns : masterwmnet: Change es2 and es3 master
operations/puppet : productionMaster switchover on es3 shard (eqiad) es1019 -> es1014
operations/mediawiki-config : mastermariadb: swithover es3 master (eqiad) es1019 -> es1014
operations/puppet : productionmariadb: Master switchover on es2 shard (eqiad) es1015 -> es1011
operations/mediawiki-config : mastermariadb: swithover es2 master (eqiad) es1015 -> es1011
operations/mediawiki-config : mastermariadb: Depool es1018 for mysql upgrade and restart
operations/mediawiki-config : mastermariadb: Depool es1014 for mysql restart
operations/mediawiki-config : mastermariadb: Depool es1011 for mysql restart
operations/mediawiki-config : mastermariadb: Depool es1016 for maintenance
operations/mediawiki-config : mastermariadb: depool db1017 for maintenance and general upgrade
operations/mediawiki-config : mastermariadb: Depool es1013 for maintenance and general upgrade
operations/mediawiki-config : mastermariadb: Depool es1012 for maintenance and upgrade
operations/puppet : productionmariadb: Drop ssl (tls) options from external storage servers

Event Timeline

jcrespo created this task.Nov 30 2016, 11:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 30 2016, 11:34 AM

Change 324411 had a related patch set uploaded (by Jcrespo):
mariadb: Drop ssl (tls) options from external storage servers

https://gerrit.wikimedia.org/r/324411

Change 324411 merged by Jcrespo:
mariadb: Drop ssl (tls) options from external storage servers

https://gerrit.wikimedia.org/r/324411

jcrespo claimed this task.Nov 30 2016, 12:26 PM
jcrespo raised the priority of this task from Medium to High.
jcrespo moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2016-11-30T12:29:38Z] <jynus> mysql restart and general upgrade for es2015 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T13:08:16Z] <jynus> mysql restart and general upgrade for es2019 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T13:15:51Z] <jynus> mysql restart and general upgrade for es2017 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T14:06:45Z] <jynus> mysql restart and general upgrade for es2014 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T14:42:41Z] <jynus> mysql restart and general upgrade for es2011 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T15:11:20Z] <jynus> mysql restart and general upgrade for es2012 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T15:29:10Z] <jynus> mysql restart and general upgrade for es2013 T151995

!log stopping for 24 hours cross-dc replication on shards es2,es3 codfw->eqiad (es1015, es1019)

Mentioned in SAL (#wikimedia-operations) [2016-11-30T15:50:36Z] <jynus> mysql restart and general upgrade for es2016 T151995

Mentioned in SAL (#wikimedia-operations) [2016-11-30T16:12:57Z] <jynus> mysql restart and general upgrade for es2018 T151995

Change 324481 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1012 for maintenance and upgrade

https://gerrit.wikimedia.org/r/324481

Change 324481 merged by Jcrespo:
mariadb: Depool es1012 for maintenance and upgrade

https://gerrit.wikimedia.org/r/324481

Mentioned in SAL (#wikimedia-operations) [2016-11-30T17:20:53Z] <jynus> mysql restart and general upgrade for es1012 T151995

Change 324518 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1013 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324518

Change 324518 merged by Jcrespo:
mariadb: Depool es1013 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324518

Mentioned in SAL (#wikimedia-operations) [2016-11-30T18:43:07Z] <jynus> mysql restart and general upgrade for es1013 T151995

Change 324539 had a related patch set uploaded (by Jcrespo):
mariadb: depool db1017 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324539

Change 324539 merged by Jcrespo:
mariadb: depool db1017 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324539

Mentioned in SAL (#wikimedia-operations) [2016-11-30T20:07:22Z] <jynus> mysql restart and general upgrade for es1017 T151995

Change 324665 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1016 for maintenance

https://gerrit.wikimedia.org/r/324665

Change 324665 merged by Jcrespo:
mariadb: Depool es1016 for maintenance

https://gerrit.wikimedia.org/r/324665

Mentioned in SAL (#wikimedia-operations) [2016-12-01T07:22:00Z] <jynus> mysql upgrade and restart for es1016 T151995

Change 324667 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1011 for mysql restart

https://gerrit.wikimedia.org/r/324667

Change 324667 merged by Jcrespo:
mariadb: Depool es1011 for mysql restart

https://gerrit.wikimedia.org/r/324667

Mentioned in SAL (#wikimedia-operations) [2016-12-01T07:38:04Z] <jynus> mysql restart for es1011 T151995

Change 324670 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1014 for mysql restart

https://gerrit.wikimedia.org/r/324670

Change 324670 merged by Jcrespo:
mariadb: Depool es1014 for mysql restart

https://gerrit.wikimedia.org/r/324670

Mentioned in SAL (#wikimedia-operations) [2016-12-01T07:51:35Z] <jynus> mysql restart for es1014 T151995

Change 324674 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1018 for mysql upgrade and restart

https://gerrit.wikimedia.org/r/324674

Change 324674 merged by Jcrespo:
mariadb: Depool es1018 for mysql upgrade and restart

https://gerrit.wikimedia.org/r/324674

Mentioned in SAL (#wikimedia-operations) [2016-12-01T09:14:19Z] <jynus> mysql restart and upgrade for es1018 T151995

Change 324683 had a related patch set uploaded (by Jcrespo):
mariadb: Master switchover on es2 shard (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324683

Change 324684 had a related patch set uploaded (by Jcrespo):
Master switchover on es3 shard (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324684

Change 324688 had a related patch set uploaded (by Jcrespo):
mariadb: swithover es2 master (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324688

Change 324688 merged by jenkins-bot:
mariadb: swithover es2 master (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324688

Change 324683 merged by Jcrespo:
mariadb: Master switchover on es2 shard (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324683

Change 324695 had a related patch set uploaded (by Marostegui):
wmnet: Change es2 master

https://gerrit.wikimedia.org/r/324695

Change 324723 had a related patch set uploaded (by Jcrespo):
mariadb: swithover es3 master (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324723

Change 324723 merged by Jcrespo:
mariadb: swithover es3 master (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324723

Change 324684 merged by Jcrespo:
Master switchover on es3 shard (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324684

Change 324695 merged by Marostegui:
wmnet: Change es2 and es3 master

https://gerrit.wikimedia.org/r/324695

Mentioned in SAL (#wikimedia-operations) [2016-12-01T15:08:34Z] <marostegui> DNS change for es2 and es3 after the master switchovers - T151995

Change 324740 had a related patch set uploaded (by Jcrespo):
Depool es1015 for maintenance

https://gerrit.wikimedia.org/r/324740

Change 324740 merged by Jcrespo:
Depool es1015 for maintenance

https://gerrit.wikimedia.org/r/324740

Mentioned in SAL (#wikimedia-operations) [2016-12-01T16:19:13Z] <jynus> mysql restart for es1015 T151995

Mentioned in SAL (#wikimedia-operations) [2016-12-01T16:58:31Z] <jynus> mysql restart for es1019 T151995

Change 324754 had a related patch set uploaded (by Jcrespo):
Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/324754

Waiting for es2019 and es2015 to warmup their buffer pools to repool them and I could close this.

Change 324754 merged by Jcrespo:
Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/324754

jcrespo closed this task as Resolved.Dec 1 2016, 7:04 PM