Page MenuHomePhabricator
Create Task
Maniphest T257284

Upgrade es4 to debian buster + mariadb 10.4
Closed, ResolvedPublic
Actions

Description

I'll start with the codfw hosts, and work my way up to doing the master switchover.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Kormat triaged this task as Medium priority.Jul 7 2020, 7:28 AM
Kormat moved this task from Unsorted 💣 to Active 🚁 on the User-Kormat board.
Kormat moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Jul 7 2020, 7:37 AM

Change 609965 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2021: disable notifications

https://gerrit.wikimedia.org/r/609965

gerritbot added a project: Patch-For-Review.Jul 7 2020, 7:37 AM

Change 609965 merged by Kormat:
[operations/puppet@production] es2021: disable notifications

https://gerrit.wikimedia.org/r/609965

gerritbot added a comment.Jul 7 2020, 7:40 AM

Change 609966 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2021 to buster.

https://gerrit.wikimedia.org/r/609966

gerritbot added a comment.Jul 7 2020, 7:42 AM

Change 609966 merged by Kormat:
[operations/puppet@production] install_server: Switch es2021 to buster.

https://gerrit.wikimedia.org/r/609966

Stashbot added a comment.Jul 7 2020, 8:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-07T08:09:15Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es2021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11767 and previous config saved to /var/cache/conftool/dbconfig/20200707-080914-kormat.json

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2020, 8:10 AM
ops-monitoring-bot added a comment.Jul 7 2020, 8:12 AM

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['es2021.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007070812_kormat_5465.log.

ops-monitoring-bot added a comment.Jul 7 2020, 8:40 AM

Completed auto-reimage of hosts:

['es2021.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Jul 7 2020, 8:56 AM

Change 609977 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2021: Re-enable notifications.

https://gerrit.wikimedia.org/r/609977

gerritbot added a project: Patch-For-Review.Jul 7 2020, 8:56 AM
gerritbot added a comment.Jul 7 2020, 9:30 AM

Change 609977 merged by Kormat:
[operations/puppet@production] es2021: Re-enable notifications.

https://gerrit.wikimedia.org/r/609977

Stashbot added a comment.Jul 7 2020, 9:40 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-07T09:40:17Z] <kormat@cumin1001> dbctl commit (dc=all): 'Repool es2021 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11774 and previous config saved to /var/cache/conftool/dbconfig/20200707-094017-kormat.json

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2020, 10:10 AM
gerritbot added a comment.Jul 7 2020, 12:04 PM

Change 610038 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Promote es2021 to es5 master in codfw

https://gerrit.wikimedia.org/r/610038

gerritbot added a project: Patch-For-Review.Jul 7 2020, 12:04 PM
Stashbot added a comment.Jul 7 2020, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-07T12:30:03Z] <kormat@cumin1001> dbctl commit (dc=all): 'Set es2021 to weight 50 T257284', diff saved to https://phabricator.wikimedia.org/P11787 and previous config saved to /var/cache/conftool/dbconfig/20200707-123003-kormat.json

gerritbot added a comment.Jul 7 2020, 12:44 PM

Change 610038 merged by Kormat:
[operations/puppet@production] mariadb: Promote es2021 to es5 master in codfw

https://gerrit.wikimedia.org/r/610038

Stashbot added a comment.Jul 7 2020, 12:44 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-07T12:44:38Z] <kormat> starting (codfw) es5 failover from es2020 to es2021 T257284

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2020, 1:10 PM
Stashbot added a comment.Jul 7 2020, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-07T13:15:24Z] <kormat@cumin1001> dbctl commit (dc=all): 'Promote es2021 to es4 master T257284', diff saved to https://phabricator.wikimedia.org/P11789 and previous config saved to /var/cache/conftool/dbconfig/20200707-131524-kormat.json

gerritbot added a comment.Jul 7 2020, 1:36 PM

Change 610061 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2020: Disable notifications

https://gerrit.wikimedia.org/r/610061

gerritbot added a project: Patch-For-Review.Jul 7 2020, 1:36 PM
gerritbot added a comment.Jul 7 2020, 1:46 PM

Change 610061 merged by Kormat:
[operations/puppet@production] es2020: Disable notifications

https://gerrit.wikimedia.org/r/610061

Kormat added a comment.EditedJul 7 2020, 1:48 PM

Current status:

  • es2021 has been reimaged
  • es2022 had already been reimaged
  • es2020 has been replaced by es2021 as codfw master for es4
  • es2020 has replication and notifications disabled, and will be reimaged tomorrow morning
Maintenance_bot removed a project: Patch-For-Review.Jul 7 2020, 2:10 PM
Stashbot added a comment.Jul 8 2020, 7:57 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-08T07:57:57Z] <kormat> reimaging es2020 to buster T257284

gerritbot added a comment.Jul 8 2020, 7:58 AM

Change 610235 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2020 to buster

https://gerrit.wikimedia.org/r/610235

gerritbot added a project: Patch-For-Review.Jul 8 2020, 7:58 AM
gerritbot added a comment.Jul 8 2020, 8:09 AM

Change 610235 merged by Kormat:
[operations/puppet@production] install_server: Switch es2020 to buster

https://gerrit.wikimedia.org/r/610235

Stashbot added a comment.Jul 8 2020, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-08T08:15:20Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es2020 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11804 and previous config saved to /var/cache/conftool/dbconfig/20200708-081519-kormat.json

ops-monitoring-bot added a comment.Jul 8 2020, 8:16 AM

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007080816_kormat_1536.log.

ops-monitoring-bot added a comment.Jul 8 2020, 9:25 AM

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

Of which those FAILED:

['es2020.codfw.wmnet']
ops-monitoring-bot added a comment.Jul 8 2020, 9:37 AM

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007080937_kormat_7040.log.

ops-monitoring-bot added a comment.Jul 8 2020, 10:04 AM

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 10 2020, 7:43 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-10T07:43:27Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1021 T257284', diff saved to https://phabricator.wikimedia.org/P11844 and previous config saved to /var/cache/conftool/dbconfig/20200710-074326-kormat.json

Stashbot added a comment.Jul 10 2020, 7:44 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-10T07:44:14Z] <kormat> reimaging es1021 to buster T257284

gerritbot added a comment.Jul 10 2020, 7:45 AM

Change 611183 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1021: Disable notifications

https://gerrit.wikimedia.org/r/611183

Stashbot added a comment.Jul 10 2020, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-10T08:01:33Z] <kormat@cumin1001> dbctl commit (dc=all): 'Reset es2020/es2021 to correct weights after master switch T257284', diff saved to https://phabricator.wikimedia.org/P11848 and previous config saved to /var/cache/conftool/dbconfig/20200710-080133-kormat.json

gerritbot added a comment.Jul 10 2020, 8:02 AM

Change 611183 merged by Kormat:
[operations/puppet@production] es1021: Disable notifications

https://gerrit.wikimedia.org/r/611183

gerritbot added a comment.Jul 10 2020, 8:06 AM

Change 611193 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es1021 to buster

https://gerrit.wikimedia.org/r/611193

Stashbot added a comment.Jul 10 2020, 8:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-10T08:08:43Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11849 and previous config saved to /var/cache/conftool/dbconfig/20200710-080843-kormat.json

gerritbot added a comment.Jul 10 2020, 8:13 AM

Change 611193 merged by Kormat:
[operations/puppet@production] install_server: Switch es1021 to buster

https://gerrit.wikimedia.org/r/611193

ops-monitoring-bot added a comment.Jul 10 2020, 8:26 AM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007100825_kormat_966.log.

ops-monitoring-bot added a comment.Jul 10 2020, 8:41 AM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007100840_kormat_15124.log.

ops-monitoring-bot added a comment.Jul 10 2020, 9:09 AM

Completed auto-reimage of hosts:

['es1021.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 10 2020, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-10T09:49:55Z] <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1021 after reimage @ 50% T257284', diff saved to https://phabricator.wikimedia.org/P11858 and previous config saved to /var/cache/conftool/dbconfig/20200710-094954-kormat.json

Stashbot added a comment.Jul 10 2020, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-10T10:21:48Z] <kormat@cumin1001> dbctl commit (dc=all): 'Finish repooling es1021, and remove weight from es1010 T257284', diff saved to https://phabricator.wikimedia.org/P11859 and previous config saved to /var/cache/conftool/dbconfig/20200710-102147-kormat.json

Kormat added a comment.Jul 10 2020, 10:22 AM

The codfw nodes are done, as is es1021. On monday i'll upgrade es1022, and then we'll need to look at scheduling a master switchover.

Stashbot added a comment.Jul 13 2020, 8:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:20:54Z] <kormat> reimaging es1022 T257284

gerritbot added a comment.Jul 13 2020, 8:24 AM

Change 612147 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1022: Disable notifications for reimaging

https://gerrit.wikimedia.org/r/612147

Stashbot added a comment.Jul 13 2020, 8:34 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:34:14Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json

gerritbot added a comment.Jul 13 2020, 8:38 AM

Change 612150 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2022 to buster

https://gerrit.wikimedia.org/r/612150

Stashbot added a comment.Jul 13 2020, 8:44 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:44:50Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json

gerritbot added a comment.Jul 13 2020, 8:45 AM

Change 612147 merged by Kormat:
[operations/puppet@production] es1022: Disable notifications for reimaging

https://gerrit.wikimedia.org/r/612147

gerritbot added a comment.Jul 13 2020, 8:45 AM

Change 612150 merged by Kormat:
[operations/puppet@production] install_server: Switch es2022 to buster

https://gerrit.wikimedia.org/r/612150

ops-monitoring-bot added a comment.Jul 13 2020, 8:57 AM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1022.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007130856_kormat_745.log.

ops-monitoring-bot added a comment.Jul 13 2020, 9:47 AM

Completed auto-reimage of hosts:

['es1022.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 13 2020, 12:08 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-13T12:08:19Z] <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json

Stashbot added a comment.Jul 13 2020, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2020-07-13T13:05:32Z] <kormat@cumin1001> dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json

gerritbot added a comment.Jul 13 2020, 2:44 PM

Change 612357 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1021: Mark as candidate master for es4

https://gerrit.wikimedia.org/r/612357

gerritbot added a comment.Jul 13 2020, 2:58 PM

Change 612357 merged by Kormat:
[operations/puppet@production] es1021: Mark as candidate master for es4

https://gerrit.wikimedia.org/r/612357

Kormat moved this task from Active 🚁 to Outgoing on the User-Kormat board.Jul 15 2020, 7:37 AM
Kormat moved this task from Outgoing to Active 🚁 on the User-Kormat board.
Kormat closed subtask T257847: Switchover es4 master from es1020 to es1021 as Resolved.Jul 21 2020, 7:35 AM
gerritbot added a comment.Jul 21 2020, 7:43 AM

Change 615147 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1020: Disable notifications for reimaging.

https://gerrit.wikimedia.org/r/615147

gerritbot added a comment.Jul 21 2020, 7:44 AM

Change 615148 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es1020 to buster.

https://gerrit.wikimedia.org/r/615148

gerritbot added a comment.Jul 21 2020, 7:46 AM

Change 615147 merged by Kormat:
[operations/puppet@production] es1020: Disable notifications for reimaging.

https://gerrit.wikimedia.org/r/615147

gerritbot added a comment.Jul 21 2020, 7:46 AM

Change 615148 merged by Kormat:
[operations/puppet@production] install_server: Switch es1020 to buster.

https://gerrit.wikimedia.org/r/615148

ops-monitoring-bot added a comment.Jul 21 2020, 7:54 AM

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007210754_kormat_19024.log.

ops-monitoring-bot added a comment.Jul 21 2020, 8:36 AM

Completed auto-reimage of hosts:

['es1020.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 21 2020, 9:21 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-21T09:21:27Z] <kormat@cumin1001> dbctl commit (dc=all): 'Re-pool es1020 at 25% in es4 T257284', diff saved to https://phabricator.wikimedia.org/P11982 and previous config saved to /var/cache/conftool/dbconfig/20200721-092126-kormat.json

Kormat closed this task as Resolved.Jul 21 2020, 9:53 AM

All of es4 is now running buster.

Stashbot added a comment.Jul 22 2020, 7:53 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T07:53:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 50% pooled in es4 T257284', diff saved to https://phabricator.wikimedia.org/P12006 and previous config saved to /var/cache/conftool/dbconfig/20200722-075312-kormat.json

Stashbot added a comment.Jul 22 2020, 8:14 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:14:57Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 75% pooled in es4, reduce es1021 to weight 25 T257284', diff saved to https://phabricator.wikimedia.org/P12009 and previous config saved to /var/cache/conftool/dbconfig/20200722-081457-kormat.json

Stashbot added a comment.Jul 22 2020, 8:42 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:41:59Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 100% pooled in es4, reduce es1021 to weight 0 T257284', diff saved to https://phabricator.wikimedia.org/P12016 and previous config saved to /var/cache/conftool/dbconfig/20200722-084159-kormat.json

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL