Page MenuHomePhabricator
Create Task
Maniphest T151995

Rolling restart of external storage servers for TLS certificate update
Closed, ResolvedPublic
Actions

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Nov 30 2016, 11:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 30 2016, 11:34 AM
gerritbot added a comment.Nov 30 2016, 11:36 AM

Change 324411 had a related patch set uploaded (by Jcrespo):
mariadb: Drop ssl (tls) options from external storage servers

https://gerrit.wikimedia.org/r/324411

gerritbot added a project: Patch-For-Review.Nov 30 2016, 11:36 AM
gerritbot added a comment.Nov 30 2016, 12:17 PM

Change 324411 merged by Jcrespo:
mariadb: Drop ssl (tls) options from external storage servers

https://gerrit.wikimedia.org/r/324411

jcrespo claimed this task.Nov 30 2016, 12:26 PM
jcrespo raised the priority of this task from Medium to High.
jcrespo moved this task from Triage to In progress on the DBA board.
Stashbot subscribed.Nov 30 2016, 12:29 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T12:29:38Z] <jynus> mysql restart and general upgrade for es2015 T151995

Stashbot added a comment.Nov 30 2016, 1:08 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T13:08:16Z] <jynus> mysql restart and general upgrade for es2019 T151995

Stashbot added a comment.Nov 30 2016, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T13:15:51Z] <jynus> mysql restart and general upgrade for es2017 T151995

Stashbot added a comment.Nov 30 2016, 2:06 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T14:06:45Z] <jynus> mysql restart and general upgrade for es2014 T151995

Stashbot added a comment.Nov 30 2016, 2:42 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T14:42:41Z] <jynus> mysql restart and general upgrade for es2011 T151995

Stashbot added a comment.Nov 30 2016, 3:11 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T15:11:20Z] <jynus> mysql restart and general upgrade for es2012 T151995

Stashbot added a comment.Nov 30 2016, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T15:29:10Z] <jynus> mysql restart and general upgrade for es2013 T151995

jcrespo added a comment.Nov 30 2016, 3:46 PM

!log stopping for 24 hours cross-dc replication on shards es2,es3 codfw->eqiad (es1015, es1019)

Stashbot added a comment.Nov 30 2016, 3:50 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T15:50:36Z] <jynus> mysql restart and general upgrade for es2016 T151995

Stashbot added a comment.Nov 30 2016, 4:12 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T16:12:57Z] <jynus> mysql restart and general upgrade for es2018 T151995

gerritbot added a comment.Nov 30 2016, 4:35 PM

Change 324481 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1012 for maintenance and upgrade

https://gerrit.wikimedia.org/r/324481

gerritbot added a comment.Nov 30 2016, 4:59 PM

Change 324481 merged by Jcrespo:
mariadb: Depool es1012 for maintenance and upgrade

https://gerrit.wikimedia.org/r/324481

Stashbot added a comment.Nov 30 2016, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T17:20:53Z] <jynus> mysql restart and general upgrade for es1012 T151995

gerritbot added a comment.Nov 30 2016, 6:24 PM

Change 324518 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1013 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324518

gerritbot added a comment.Nov 30 2016, 6:36 PM

Change 324518 merged by Jcrespo:
mariadb: Depool es1013 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324518

Stashbot added a comment.Nov 30 2016, 6:43 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T18:43:07Z] <jynus> mysql restart and general upgrade for es1013 T151995

gerritbot added a comment.Nov 30 2016, 7:19 PM

Change 324539 had a related patch set uploaded (by Jcrespo):
mariadb: depool db1017 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324539

gerritbot added a comment.Nov 30 2016, 7:49 PM

Change 324539 merged by Jcrespo:
mariadb: depool db1017 for maintenance and general upgrade

https://gerrit.wikimedia.org/r/324539

Stashbot added a comment.Nov 30 2016, 8:07 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-30T20:07:22Z] <jynus> mysql restart and general upgrade for es1017 T151995

gerritbot added a comment.Dec 1 2016, 7:08 AM

Change 324665 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1016 for maintenance

https://gerrit.wikimedia.org/r/324665

gerritbot added a comment.Dec 1 2016, 7:10 AM

Change 324665 merged by Jcrespo:
mariadb: Depool es1016 for maintenance

https://gerrit.wikimedia.org/r/324665

Stashbot added a comment.Dec 1 2016, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T07:22:00Z] <jynus> mysql upgrade and restart for es1016 T151995

gerritbot added a comment.Dec 1 2016, 7:28 AM

Change 324667 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1011 for mysql restart

https://gerrit.wikimedia.org/r/324667

gerritbot added a comment.Dec 1 2016, 7:33 AM

Change 324667 merged by Jcrespo:
mariadb: Depool es1011 for mysql restart

https://gerrit.wikimedia.org/r/324667

Stashbot added a comment.Dec 1 2016, 7:38 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T07:38:04Z] <jynus> mysql restart for es1011 T151995

gerritbot added a comment.Dec 1 2016, 7:46 AM

Change 324670 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1014 for mysql restart

https://gerrit.wikimedia.org/r/324670

gerritbot added a comment.Dec 1 2016, 7:48 AM

Change 324670 merged by Jcrespo:
mariadb: Depool es1014 for mysql restart

https://gerrit.wikimedia.org/r/324670

Stashbot added a comment.Dec 1 2016, 7:51 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T07:51:35Z] <jynus> mysql restart for es1014 T151995

gerritbot added a comment.Dec 1 2016, 8:08 AM

Change 324674 had a related patch set uploaded (by Jcrespo):
mariadb: Depool es1018 for mysql upgrade and restart

https://gerrit.wikimedia.org/r/324674

gerritbot added a comment.Dec 1 2016, 9:05 AM

Change 324674 merged by Jcrespo:
mariadb: Depool es1018 for mysql upgrade and restart

https://gerrit.wikimedia.org/r/324674

Stashbot added a comment.Dec 1 2016, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T09:14:19Z] <jynus> mysql restart and upgrade for es1018 T151995

gerritbot added a comment.Dec 1 2016, 9:27 AM

Change 324683 had a related patch set uploaded (by Jcrespo):
mariadb: Master switchover on es2 shard (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324683

gerritbot added a comment.Dec 1 2016, 9:28 AM

Change 324684 had a related patch set uploaded (by Jcrespo):
Master switchover on es3 shard (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324684

gerritbot added a comment.Dec 1 2016, 10:04 AM

Change 324688 had a related patch set uploaded (by Jcrespo):
mariadb: swithover es2 master (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324688

gerritbot added a comment.Dec 1 2016, 10:10 AM

Change 324688 merged by jenkins-bot:
mariadb: swithover es2 master (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324688

gerritbot added a comment.Dec 1 2016, 10:17 AM

Change 324683 merged by Jcrespo:
mariadb: Master switchover on es2 shard (eqiad) es1015 -> es1011

https://gerrit.wikimedia.org/r/324683

gerritbot added a comment.Dec 1 2016, 10:28 AM

Change 324695 had a related patch set uploaded (by Marostegui):
wmnet: Change es2 master

https://gerrit.wikimedia.org/r/324695

gerritbot added a comment.Dec 1 2016, 2:17 PM

Change 324723 had a related patch set uploaded (by Jcrespo):
mariadb: swithover es3 master (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324723

gerritbot added a comment.Dec 1 2016, 2:44 PM

Change 324723 merged by Jcrespo:
mariadb: swithover es3 master (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324723

gerritbot added a comment.Dec 1 2016, 2:49 PM

Change 324684 merged by Jcrespo:
Master switchover on es3 shard (eqiad) es1019 -> es1014

https://gerrit.wikimedia.org/r/324684

gerritbot added a comment.Dec 1 2016, 3:05 PM

Change 324695 merged by Marostegui:
wmnet: Change es2 and es3 master

https://gerrit.wikimedia.org/r/324695

Stashbot added a comment.Dec 1 2016, 3:08 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T15:08:34Z] <marostegui> DNS change for es2 and es3 after the master switchovers - T151995

gerritbot added a comment.Dec 1 2016, 4:10 PM

Change 324740 had a related patch set uploaded (by Jcrespo):
Depool es1015 for maintenance

https://gerrit.wikimedia.org/r/324740

gerritbot added a comment.Dec 1 2016, 4:10 PM

Change 324740 merged by Jcrespo:
Depool es1015 for maintenance

https://gerrit.wikimedia.org/r/324740

Stashbot added a comment.Dec 1 2016, 4:19 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T16:19:13Z] <jynus> mysql restart for es1015 T151995

Stashbot added a comment.Dec 1 2016, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2016-12-01T16:58:31Z] <jynus> mysql restart for es1019 T151995

gerritbot added a comment.Dec 1 2016, 5:01 PM

Change 324754 had a related patch set uploaded (by Jcrespo):
Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/324754

jcrespo added a comment.Dec 1 2016, 5:10 PM

Waiting for es2019 and es2015 to warmup their buffer pools to repool them and I could close this.

gerritbot added a comment.Dec 1 2016, 6:48 PM

Change 324754 merged by Jcrespo:
Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/324754

jcrespo closed this task as Resolved.Dec 1 2016, 7:04 PM
AndyRussG mentioned this in T152122: Central Notice: possible CN issue early on December 1st UTC.Dec 1 2016, 9:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL