I'll start with the codfw hosts, and work my way up to doing the master switchover.
Description
Details
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Stalled | None | T302086 Set scap minimum python version to 3.7 | |||
Resolved | None | T247045 Migrate all of production metal and VMs to Buster or later | |||
Resolved | • Marostegui | T250666 Upgrade WMF database-and-backup-related hosts to buster | |||
Resolved | Kormat | T257284 Upgrade es4 to debian buster + mariadb 10.4 | |||
Resolved | Kormat | T257847 Switchover es4 master from es1020 to es1021 |
Event Timeline
Change 609965 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2021: disable notifications
Change 609965 merged by Kormat:
[operations/puppet@production] es2021: disable notifications
Change 609966 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2021 to buster.
Change 609966 merged by Kormat:
[operations/puppet@production] install_server: Switch es2021 to buster.
Mentioned in SAL (#wikimedia-operations) [2020-07-07T08:09:15Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es2021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11767 and previous config saved to /var/cache/conftool/dbconfig/20200707-080914-kormat.json
Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:
['es2021.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007070812_kormat_5465.log.
Change 609977 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2021: Re-enable notifications.
Change 609977 merged by Kormat:
[operations/puppet@production] es2021: Re-enable notifications.
Mentioned in SAL (#wikimedia-operations) [2020-07-07T09:40:17Z] <kormat@cumin1001> dbctl commit (dc=all): 'Repool es2021 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11774 and previous config saved to /var/cache/conftool/dbconfig/20200707-094017-kormat.json
Change 610038 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Promote es2021 to es5 master in codfw
Mentioned in SAL (#wikimedia-operations) [2020-07-07T12:30:03Z] <kormat@cumin1001> dbctl commit (dc=all): 'Set es2021 to weight 50 T257284', diff saved to https://phabricator.wikimedia.org/P11787 and previous config saved to /var/cache/conftool/dbconfig/20200707-123003-kormat.json
Change 610038 merged by Kormat:
[operations/puppet@production] mariadb: Promote es2021 to es5 master in codfw
Mentioned in SAL (#wikimedia-operations) [2020-07-07T12:44:38Z] <kormat> starting (codfw) es5 failover from es2020 to es2021 T257284
Mentioned in SAL (#wikimedia-operations) [2020-07-07T13:15:24Z] <kormat@cumin1001> dbctl commit (dc=all): 'Promote es2021 to es4 master T257284', diff saved to https://phabricator.wikimedia.org/P11789 and previous config saved to /var/cache/conftool/dbconfig/20200707-131524-kormat.json
Change 610061 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2020: Disable notifications
Change 610061 merged by Kormat:
[operations/puppet@production] es2020: Disable notifications
Current status:
- es2021 has been reimaged
- es2022 had already been reimaged
- es2020 has been replaced by es2021 as codfw master for es4
- es2020 has replication and notifications disabled, and will be reimaged tomorrow morning
Mentioned in SAL (#wikimedia-operations) [2020-07-08T07:57:57Z] <kormat> reimaging es2020 to buster T257284
Change 610235 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2020 to buster
Change 610235 merged by Kormat:
[operations/puppet@production] install_server: Switch es2020 to buster
Mentioned in SAL (#wikimedia-operations) [2020-07-08T08:15:20Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es2020 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11804 and previous config saved to /var/cache/conftool/dbconfig/20200708-081519-kormat.json
Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:
['es2020.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007080816_kormat_1536.log.
Completed auto-reimage of hosts:
['es2020.codfw.wmnet']
Of which those FAILED:
['es2020.codfw.wmnet']
Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:
['es2020.codfw.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007080937_kormat_7040.log.
Mentioned in SAL (#wikimedia-operations) [2020-07-10T07:43:27Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1021 T257284', diff saved to https://phabricator.wikimedia.org/P11844 and previous config saved to /var/cache/conftool/dbconfig/20200710-074326-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-07-10T07:44:14Z] <kormat> reimaging es1021 to buster T257284
Change 611183 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1021: Disable notifications
Mentioned in SAL (#wikimedia-operations) [2020-07-10T08:01:33Z] <kormat@cumin1001> dbctl commit (dc=all): 'Reset es2020/es2021 to correct weights after master switch T257284', diff saved to https://phabricator.wikimedia.org/P11848 and previous config saved to /var/cache/conftool/dbconfig/20200710-080133-kormat.json
Change 611183 merged by Kormat:
[operations/puppet@production] es1021: Disable notifications
Change 611193 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es1021 to buster
Mentioned in SAL (#wikimedia-operations) [2020-07-10T08:08:43Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11849 and previous config saved to /var/cache/conftool/dbconfig/20200710-080843-kormat.json
Change 611193 merged by Kormat:
[operations/puppet@production] install_server: Switch es1021 to buster
Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:
['es1021.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007100825_kormat_966.log.
Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:
['es1021.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007100840_kormat_15124.log.
Mentioned in SAL (#wikimedia-operations) [2020-07-10T09:49:55Z] <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1021 after reimage @ 50% T257284', diff saved to https://phabricator.wikimedia.org/P11858 and previous config saved to /var/cache/conftool/dbconfig/20200710-094954-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-07-10T10:21:48Z] <kormat@cumin1001> dbctl commit (dc=all): 'Finish repooling es1021, and remove weight from es1010 T257284', diff saved to https://phabricator.wikimedia.org/P11859 and previous config saved to /var/cache/conftool/dbconfig/20200710-102147-kormat.json
The codfw nodes are done, as is es1021. On monday i'll upgrade es1022, and then we'll need to look at scheduling a master switchover.
Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:20:54Z] <kormat> reimaging es1022 T257284
Change 612147 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1022: Disable notifications for reimaging
Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:34:14Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json
Change 612150 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2022 to buster
Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:44:50Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json
Change 612147 merged by Kormat:
[operations/puppet@production] es1022: Disable notifications for reimaging
Change 612150 merged by Kormat:
[operations/puppet@production] install_server: Switch es2022 to buster
Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:
['es1022.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007130856_kormat_745.log.
Mentioned in SAL (#wikimedia-operations) [2020-07-13T12:08:19Z] <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-07-13T13:05:32Z] <kormat@cumin1001> dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json
Change 612357 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1021: Mark as candidate master for es4
Change 612357 merged by Kormat:
[operations/puppet@production] es1021: Mark as candidate master for es4
Change 615147 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1020: Disable notifications for reimaging.
Change 615148 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es1020 to buster.
Change 615147 merged by Kormat:
[operations/puppet@production] es1020: Disable notifications for reimaging.
Change 615148 merged by Kormat:
[operations/puppet@production] install_server: Switch es1020 to buster.
Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:
['es1020.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202007210754_kormat_19024.log.
Mentioned in SAL (#wikimedia-operations) [2020-07-21T09:21:27Z] <kormat@cumin1001> dbctl commit (dc=all): 'Re-pool es1020 at 25% in es4 T257284', diff saved to https://phabricator.wikimedia.org/P11982 and previous config saved to /var/cache/conftool/dbconfig/20200721-092126-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-07-22T07:53:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 50% pooled in es4 T257284', diff saved to https://phabricator.wikimedia.org/P12006 and previous config saved to /var/cache/conftool/dbconfig/20200722-075312-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:14:57Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 75% pooled in es4, reduce es1021 to weight 25 T257284', diff saved to https://phabricator.wikimedia.org/P12009 and previous config saved to /var/cache/conftool/dbconfig/20200722-081457-kormat.json
Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:41:59Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 100% pooled in es4, reduce es1021 to weight 0 T257284', diff saved to https://phabricator.wikimedia.org/P12016 and previous config saved to /var/cache/conftool/dbconfig/20200722-084159-kormat.json