Page MenuHomePhabricator

Reimage db1134 to Buster and repool it
Closed, ResolvedPublic

Description

db1134 crashed (T274472), a different host was chosen to be candidate master (db1163).
db1134 needs to:

  • Get reimaged to Buster
  • Install 10.4.18
  • Clone db1134 from db1169
  • Get repooled

Event Timeline

Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2021-02-24T13:46:39Z] <marostegui> Compare data between db1134 and db1163 T275343

Change 666631 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db1134 to Buster

https://gerrit.wikimedia.org/r/666631

Change 666631 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db1134 to Buster

https://gerrit.wikimedia.org/r/666631

Mentioned in SAL (#wikimedia-operations) [2021-02-25T11:40:20Z] <marostegui> Stop MySQL on db1134 to reimage it to buster T275343

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1134.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202102251143_marostegui_5186.log.

Completed auto-reimage of hosts:

['db1134.eqiad.wmnet']

and were ALL successful.

I will start repooling this host on Monday

I will start repooling this host on Monday

This host crashed overnight, so the data is probably corrupted from the previous crash. So it needs to be rebuilt

Mentioned in SAL (#wikimedia-operations) [2021-02-26T06:17:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1169 to clone db1134 T275343', diff saved to https://phabricator.wikimedia.org/P14490 and previous config saved to /var/cache/conftool/dbconfig/20210226-061705-marostegui.json

db1134 has been cloned. Will leave it running during the weekend before repooling it

Change 667430 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1134: Enable notifications

https://gerrit.wikimedia.org/r/667430

Change 667430 merged by Marostegui:
[operations/puppet@production] db1134: Enable notifications

https://gerrit.wikimedia.org/r/667430

I have started to slowly repool this host

Marostegui updated the task description. (Show Details)

Host fully pooled.