Page MenuHomePhabricator

Productionize db213[6-9] and db2140
Closed, ResolvedPublic

Description

The following hosts need to be productionized in s4:

  • db2136 will replace db2073 (sanitarium master)
    • db2136 pooled in s4
    • change db2095 to replicate from db2136
  • db2137 will replace db2084 (multi instance with s4 and s5)
  • db2138 will replace db2091 (multi instance with s4 and s2)
  • db2139 will replace db2099 (multi instance with s4 and s5)
    • db2139 pooled in s4+s5
    • move backups to use db2139 instead of db2099
  • db2140 will replace db2090 (master)
    • db2140 pooled in s4
    • move codfw s4 master to db2140

Event Timeline

Marostegui triaged this task as Medium priority.May 18 2020, 5:21 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.

Change 597050 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] Add db2136 to s4

https://gerrit.wikimedia.org/r/597050

Change 597050 merged by Kormat:
[operations/puppet@production] mariadb: Add db2136 to s4

https://gerrit.wikimedia.org/r/597050

Mentioned in SAL (#wikimedia-operations) [2020-05-18T14:15:05Z] <kormat@cumin1001> dbctl commit (dc=all): 'Depool db2073 while replacing it T252985', diff saved to https://phabricator.wikimedia.org/P11216 and previous config saved to /var/cache/conftool/dbconfig/20200518-141505-kormat.json

Change 597224 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Remove db2136 from spare role.

https://gerrit.wikimedia.org/r/597224

Change 597224 merged by Kormat:
[operations/puppet@production] mariadb: Remove db2136 from spare role.

https://gerrit.wikimedia.org/r/597224

Change 597273 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] hieradata: enable notifications for db2136

https://gerrit.wikimedia.org/r/597273

Change 597273 merged by Kormat:
[operations/puppet@production] hieradata: enable notifications for db2136

https://gerrit.wikimedia.org/r/597273

Mentioned in SAL (#wikimedia-operations) [2020-05-19T13:03:14Z] <kormat@cumin1001> dbctl commit (dc=all): 'Pool db2136 into s4 T252985', diff saved to https://phabricator.wikimedia.org/P11233 and previous config saved to /var/cache/conftool/dbconfig/20200519-130313-kormat.json

Kormat updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2020-05-19T15:23:41Z] <kormat@cumin1001> dbctl commit (dc=all): 'Repool db2073 into s4 T252985', diff saved to https://phabricator.wikimedia.org/P11236 and previous config saved to /var/cache/conftool/dbconfig/20200519-152340-kormat.json

db2137 is set up in s4+s5, and it's currently restoring logical backups from dbprov2001.

Change 598061 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Enable notifications for db2137

https://gerrit.wikimedia.org/r/598061

Change 598061 merged by Kormat:
[operations/puppet@production] mariadb: Enable notifications for db2137

https://gerrit.wikimedia.org/r/598061

Mentioned in SAL (#wikimedia-operations) [2020-05-22T15:01:21Z] <kormat@cumin1001> dbctl commit (dc=all): 'Pool db2137 into s4+s5 T252985', diff saved to https://phabricator.wikimedia.org/P11292 and previous config saved to /var/cache/conftool/dbconfig/20200522-150120-kormat.json

Mentioned in SAL (#wikimedia-operations) [2020-05-25T11:27:20Z] <marostegui> Extend /srv 1100G on db213[6-9] T252985

Change 598736 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Add db2138 to s2+s4

https://gerrit.wikimedia.org/r/598736

Change 598736 merged by Kormat:
[operations/puppet@production] mariadb: Add db2138 to s2+s4

https://gerrit.wikimedia.org/r/598736

db2138 is set up, and currently catching up on replication before it gets pooled into s2+s4.

Change 599304 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Enable notifications for db2138

https://gerrit.wikimedia.org/r/599304

Change 599304 merged by Kormat:
[operations/puppet@production] mariadb: Enable notifications for db2138

https://gerrit.wikimedia.org/r/599304

Mentioned in SAL (#wikimedia-operations) [2020-05-28T11:03:34Z] <kormat@cumin1001> dbctl commit (dc=all): 'Add db2138 to s2+s4 T252985', diff saved to https://phabricator.wikimedia.org/P11330 and previous config saved to /var/cache/conftool/dbconfig/20200528-110333-kormat.json

Change 599313 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Add db2139 to s4+s5

https://gerrit.wikimedia.org/r/599313

Change 599313 merged by Kormat:
[operations/puppet@production] mariadb: Add db2139 to s4+s5

https://gerrit.wikimedia.org/r/599313

Change 599739 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Enable notifications for db2139

https://gerrit.wikimedia.org/r/599739

Change 599746 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] maraidb: Add db2040 to s4

https://gerrit.wikimedia.org/r/599746

Change 599739 merged by Kormat:
[operations/puppet@production] mariadb: Enable notifications for db2139

https://gerrit.wikimedia.org/r/599739

Change 599746 merged by Kormat:
[operations/puppet@production] mariadb: Add db2140 to s4

https://gerrit.wikimedia.org/r/599746

Change 601704 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/puppet@production] mariadb: Enable notifications for db2140

https://gerrit.wikimedia.org/r/601704

Mentioned in SAL (#wikimedia-operations) [2020-06-02T12:30:21Z] <kormat@cumin1001> dbctl commit (dc=all): 'Repool db2110, copy to db2140 complete T252985', diff saved to https://phabricator.wikimedia.org/P11362 and previous config saved to /var/cache/conftool/dbconfig/20200602-123020-kormat.json

Change 601704 merged by Kormat:
[operations/puppet@production] mariadb: Enable notifications for db2140

https://gerrit.wikimedia.org/r/601704

Mentioned in SAL (#wikimedia-operations) [2020-06-02T12:50:12Z] <kormat@cumin1001> dbctl commit (dc=all): 'Pool db2140 into s4 T252985', diff saved to https://phabricator.wikimedia.org/P11363 and previous config saved to /var/cache/conftool/dbconfig/20200602-125012-kormat.json

Marking as resolved - the remaining bits are blocked by labsdb not being on 10.4 yet, and will be taken care of as part of decommissioning the old hosts once that block is cleared.