Page MenuHomePhabricator

Upgrade es4 to debian buster + mariadb 10.4
Closed, ResolvedPublic

Description

I'll start with the codfw hosts, and work my way up to doing the master switchover.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Kormat triaged this task as Medium priority.Jul 7 2020, 7:28 AM
Kormat moved this task from Unsorted 💣 to Active 🚁 on the User-Kormat board.
Kormat moved this task from Triage to In progress on the DBA board.

Change 609965 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2021: disable notifications

https://gerrit.wikimedia.org/r/609965

Change 609965 merged by Kormat:
[operations/puppet@production] es2021: disable notifications

https://gerrit.wikimedia.org/r/609965

Change 609966 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2021 to buster.

https://gerrit.wikimedia.org/r/609966

Change 609966 merged by Kormat:
[operations/puppet@production] install_server: Switch es2021 to buster.

https://gerrit.wikimedia.org/r/609966

Mentioned in SAL (#wikimedia-operations) [2020-07-07T08:09:15Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es2021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11767 and previous config saved to /var/cache/conftool/dbconfig/20200707-080914-kormat.json

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['es2021.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007070812_kormat_5465.log.

Completed auto-reimage of hosts:

['es2021.codfw.wmnet']

and were ALL successful.

Change 609977 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2021: Re-enable notifications.

https://gerrit.wikimedia.org/r/609977

Change 609977 merged by Kormat:
[operations/puppet@production] es2021: Re-enable notifications.

https://gerrit.wikimedia.org/r/609977

Mentioned in SAL (#wikimedia-operations) [2020-07-07T09:40:17Z] <kormat@cumin1001> dbctl commit (dc=all): 'Repool es2021 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11774 and previous config saved to /var/cache/conftool/dbconfig/20200707-094017-kormat.json

Change 610038 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Promote es2021 to es5 master in codfw

https://gerrit.wikimedia.org/r/610038

Mentioned in SAL (#wikimedia-operations) [2020-07-07T12:30:03Z] <kormat@cumin1001> dbctl commit (dc=all): 'Set es2021 to weight 50 T257284', diff saved to https://phabricator.wikimedia.org/P11787 and previous config saved to /var/cache/conftool/dbconfig/20200707-123003-kormat.json

Change 610038 merged by Kormat:
[operations/puppet@production] mariadb: Promote es2021 to es5 master in codfw

https://gerrit.wikimedia.org/r/610038

Mentioned in SAL (#wikimedia-operations) [2020-07-07T12:44:38Z] <kormat> starting (codfw) es5 failover from es2020 to es2021 T257284

Mentioned in SAL (#wikimedia-operations) [2020-07-07T13:15:24Z] <kormat@cumin1001> dbctl commit (dc=all): 'Promote es2021 to es4 master T257284', diff saved to https://phabricator.wikimedia.org/P11789 and previous config saved to /var/cache/conftool/dbconfig/20200707-131524-kormat.json

Change 610061 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es2020: Disable notifications

https://gerrit.wikimedia.org/r/610061

Change 610061 merged by Kormat:
[operations/puppet@production] es2020: Disable notifications

https://gerrit.wikimedia.org/r/610061

Current status:

  • es2021 has been reimaged
  • es2022 had already been reimaged
  • es2020 has been replaced by es2021 as codfw master for es4
  • es2020 has replication and notifications disabled, and will be reimaged tomorrow morning

Change 610235 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2020 to buster

https://gerrit.wikimedia.org/r/610235

Change 610235 merged by Kormat:
[operations/puppet@production] install_server: Switch es2020 to buster

https://gerrit.wikimedia.org/r/610235

Mentioned in SAL (#wikimedia-operations) [2020-07-08T08:15:20Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es2020 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11804 and previous config saved to /var/cache/conftool/dbconfig/20200708-081519-kormat.json

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007080816_kormat_1536.log.

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

Of which those FAILED:

['es2020.codfw.wmnet']

Script wmf-auto-reimage was launched by kormat on cumin2001.codfw.wmnet for hosts:

['es2020.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007080937_kormat_7040.log.

Completed auto-reimage of hosts:

['es2020.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-07-10T07:43:27Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1021 T257284', diff saved to https://phabricator.wikimedia.org/P11844 and previous config saved to /var/cache/conftool/dbconfig/20200710-074326-kormat.json

Change 611183 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1021: Disable notifications

https://gerrit.wikimedia.org/r/611183

Mentioned in SAL (#wikimedia-operations) [2020-07-10T08:01:33Z] <kormat@cumin1001> dbctl commit (dc=all): 'Reset es2020/es2021 to correct weights after master switch T257284', diff saved to https://phabricator.wikimedia.org/P11848 and previous config saved to /var/cache/conftool/dbconfig/20200710-080133-kormat.json

Change 611183 merged by Kormat:
[operations/puppet@production] es1021: Disable notifications

https://gerrit.wikimedia.org/r/611183

Change 611193 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es1021 to buster

https://gerrit.wikimedia.org/r/611193

Mentioned in SAL (#wikimedia-operations) [2020-07-10T08:08:43Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11849 and previous config saved to /var/cache/conftool/dbconfig/20200710-080843-kormat.json

Change 611193 merged by Kormat:
[operations/puppet@production] install_server: Switch es1021 to buster

https://gerrit.wikimedia.org/r/611193

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007100825_kormat_966.log.

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1021.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007100840_kormat_15124.log.

Completed auto-reimage of hosts:

['es1021.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-07-10T09:49:55Z] <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1021 after reimage @ 50% T257284', diff saved to https://phabricator.wikimedia.org/P11858 and previous config saved to /var/cache/conftool/dbconfig/20200710-094954-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-07-10T10:21:48Z] <kormat@cumin1001> dbctl commit (dc=all): 'Finish repooling es1021, and remove weight from es1010 T257284', diff saved to https://phabricator.wikimedia.org/P11859 and previous config saved to /var/cache/conftool/dbconfig/20200710-102147-kormat.json

The codfw nodes are done, as is es1021. On monday i'll upgrade es1022, and then we'll need to look at scheduling a master switchover.

Change 612147 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1022: Disable notifications for reimaging

https://gerrit.wikimedia.org/r/612147

Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:34:14Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1022 T257284', diff saved to https://phabricator.wikimedia.org/P11869 and previous config saved to /var/cache/conftool/dbconfig/20200713-083414-kormat.json

Change 612150 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es2022 to buster

https://gerrit.wikimedia.org/r/612150

Mentioned in SAL (#wikimedia-operations) [2020-07-13T08:44:50Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool es1022 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11871 and previous config saved to /var/cache/conftool/dbconfig/20200713-084449-kormat.json

Change 612147 merged by Kormat:
[operations/puppet@production] es1022: Disable notifications for reimaging

https://gerrit.wikimedia.org/r/612147

Change 612150 merged by Kormat:
[operations/puppet@production] install_server: Switch es2022 to buster

https://gerrit.wikimedia.org/r/612150

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1022.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007130856_kormat_745.log.

Completed auto-reimage of hosts:

['es1022.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-07-13T12:08:19Z] <kormat@cumin1001> dbctl commit (dc=all): 'Start repooling es1022 after reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11873 and previous config saved to /var/cache/conftool/dbconfig/20200713-120818-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-07-13T13:05:32Z] <kormat@cumin1001> dbctl commit (dc=all): 'Fully repool es1022, and set es1020 to zero weight T257284', diff saved to https://phabricator.wikimedia.org/P11878 and previous config saved to /var/cache/conftool/dbconfig/20200713-130532-kormat.json

Change 612357 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1021: Mark as candidate master for es4

https://gerrit.wikimedia.org/r/612357

Change 612357 merged by Kormat:
[operations/puppet@production] es1021: Mark as candidate master for es4

https://gerrit.wikimedia.org/r/612357

Kormat moved this task from Outgoing to Active 🚁 on the User-Kormat board.

Change 615147 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] es1020: Disable notifications for reimaging.

https://gerrit.wikimedia.org/r/615147

Change 615148 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] install_server: Switch es1020 to buster.

https://gerrit.wikimedia.org/r/615148

Change 615147 merged by Kormat:
[operations/puppet@production] es1020: Disable notifications for reimaging.

https://gerrit.wikimedia.org/r/615147

Change 615148 merged by Kormat:
[operations/puppet@production] install_server: Switch es1020 to buster.

https://gerrit.wikimedia.org/r/615148

Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts:

['es1020.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007210754_kormat_19024.log.

Completed auto-reimage of hosts:

['es1020.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-07-21T09:21:27Z] <kormat@cumin1001> dbctl commit (dc=all): 'Re-pool es1020 at 25% in es4 T257284', diff saved to https://phabricator.wikimedia.org/P11982 and previous config saved to /var/cache/conftool/dbconfig/20200721-092126-kormat.json

All of es4 is now running buster.

Mentioned in SAL (#wikimedia-operations) [2020-07-22T07:53:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 50% pooled in es4 T257284', diff saved to https://phabricator.wikimedia.org/P12006 and previous config saved to /var/cache/conftool/dbconfig/20200722-075312-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:14:57Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 75% pooled in es4, reduce es1021 to weight 25 T257284', diff saved to https://phabricator.wikimedia.org/P12009 and previous config saved to /var/cache/conftool/dbconfig/20200722-081457-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-07-22T08:41:59Z] <kormat@cumin1001> dbctl commit (dc=all): 'Increase es1020 to 100% pooled in es4, reduce es1021 to weight 0 T257284', diff saved to https://phabricator.wikimedia.org/P12016 and previous config saved to /var/cache/conftool/dbconfig/20200722-084159-kormat.json