db1134 crashed (T274472), a different host was chosen to be candidate master (db1163).
db1134 needs to:
- Get reimaged to Buster
- Install 10.4.18
- Clone db1134 from db1169
- Get repooled
db1134 crashed (T274472), a different host was chosen to be candidate master (db1163).
db1134 needs to:
Mentioned in SAL (#wikimedia-operations) [2021-02-24T13:46:39Z] <marostegui> Compare data between db1134 and db1163 T275343
Change 666631 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db1134 to Buster
Change 666631 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db1134 to Buster
Mentioned in SAL (#wikimedia-operations) [2021-02-25T11:40:20Z] <marostegui> Stop MySQL on db1134 to reimage it to buster T275343
Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:
['db1134.eqiad.wmnet']
The log can be found in /var/log/wmf-auto-reimage/202102251143_marostegui_5186.log.
This host crashed overnight, so the data is probably corrupted from the previous crash. So it needs to be rebuilt
Mentioned in SAL (#wikimedia-operations) [2021-02-26T06:17:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1169 to clone db1134 T275343', diff saved to https://phabricator.wikimedia.org/P14490 and previous config saved to /var/cache/conftool/dbconfig/20210226-061705-marostegui.json
Change 667430 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1134: Enable notifications
Change 667430 merged by Marostegui:
[operations/puppet@production] db1134: Enable notifications