Page MenuHomePhabricator

Productionize db114[1-9]
Closed, ResolvedPublic

Description

These hosts are the ones that will replace the ones in s4

  • db1141 -> db1081 (candidate master)
  • db1142 -> db1084
  • db1143 -> db1091
  • db1144 -> db1097 (multi-instance with s5 as well)
  • db1145 -> db1102 (backup source with s5 as well)
  • db1146 -> db1103 (multi-instance with s2 as well)
  • db1147 -> extra host for performance growth
  • db1148 -> db1121 (sanitarium master)
  • db1149 -> db1138 (current master)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+4 -5
operations/puppetproduction+9 -5
operations/puppetproduction+2 -1
operations/puppetproduction+1 -2
operations/puppetproduction+7 -9
operations/puppetproduction+1 -0
operations/puppetproduction+1 -1
operations/puppetproduction+5 -5
operations/puppetproduction+5 -5
operations/puppetproduction+0 -1
operations/puppetproduction+5 -2
operations/puppetproduction+0 -1
operations/puppetproduction+4 -2
operations/puppetproduction+0 -1
operations/puppetproduction+10 -4
operations/puppetproduction+0 -1
operations/puppetproduction+10 -4
operations/puppetproduction+0 -1
operations/puppetproduction+6 -2
operations/puppetproduction+0 -1
operations/puppetproduction+4 -2
operations/puppetproduction+0 -1
operations/puppetproduction+4 -2
operations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedCmjohnson
ResolvedMarostegui
StalledMarostegui

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2020-06-01T11:30:33Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11343 and previous config saved to /var/cache/conftool/dbconfig/20200601-113032-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-01T11:44:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11344 and previous config saved to /var/cache/conftool/dbconfig/20200601-114440-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-01T12:00:00Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1142, db1147 T252512', diff saved to https://phabricator.wikimedia.org/P11345 and previous config saved to /var/cache/conftool/dbconfig/20200601-120000-marostegui.json

Marostegui updated the task description. (Show Details)Jun 1 2020, 12:00 PM

Change 601641 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Place db1148 into s4

https://gerrit.wikimedia.org/r/601641

Change 601641 merged by Marostegui:
[operations/puppet@production] mariadb: Place db1148 into s4

https://gerrit.wikimedia.org/r/601641

Mentioned in SAL (#wikimedia-operations) [2020-06-02T09:38:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1148 to dbctl depooled T252512', diff saved to https://phabricator.wikimedia.org/P11356 and previous config saved to /var/cache/conftool/dbconfig/20200602-093841-marostegui.json

Change 601670 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1148: Enable notifications

https://gerrit.wikimedia.org/r/601670

Change 601670 merged by Marostegui:
[operations/puppet@production] db1148: Enable notifications

https://gerrit.wikimedia.org/r/601670

Mentioned in SAL (#wikimedia-operations) [2020-06-02T09:44:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11357 and previous config saved to /var/cache/conftool/dbconfig/20200602-094441-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-02T09:53:22Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11359 and previous config saved to /var/cache/conftool/dbconfig/20200602-095321-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-02T10:02:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11360 and previous config saved to /var/cache/conftool/dbconfig/20200602-100246-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-02T10:11:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1121, db1148 T252512', diff saved to https://phabricator.wikimedia.org/P11361 and previous config saved to /var/cache/conftool/dbconfig/20200602-101150-marostegui.json

Marostegui updated the task description. (Show Details)Jun 2 2020, 10:12 AM
Marostegui changed the status of subtask T253217: Relocate "old" s4 hosts from Stalled to Open.Jun 3 2020, 10:22 AM

Change 602042 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1091 from s4 to s1

https://gerrit.wikimedia.org/r/602042

Mentioned in SAL (#wikimedia-operations) [2020-06-03T10:42:51Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1091 - will be reimaged and moved to s1 T252512', diff saved to https://phabricator.wikimedia.org/P11381 and previous config saved to /var/cache/conftool/dbconfig/20200603-104251-marostegui.json

Change 602277 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1091 from s4 to s1

https://gerrit.wikimedia.org/r/602277

Change 602277 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1091 from s4 to s1

https://gerrit.wikimedia.org/r/602277

Change 602042 abandoned by Marostegui:
mariadb: Move db1091 from s4 to s1

Reason:
Done via 602277

https://gerrit.wikimedia.org/r/602042

Change 603338 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Allow reimage db1141

https://gerrit.wikimedia.org/r/603338

Change 603338 merged by Marostegui:
[operations/puppet@production] install_server: Allow reimage db1141

https://gerrit.wikimedia.org/r/603338

Change 603419 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1141: Disable notifications

https://gerrit.wikimedia.org/r/603419

Change 603419 merged by Marostegui:
[operations/puppet@production] db1141: Disable notifications

https://gerrit.wikimedia.org/r/603419

Change 603907 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1141 from labsdb role to s4

https://gerrit.wikimedia.org/r/603907

Change 603907 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1141 from labsdb role to s4

https://gerrit.wikimedia.org/r/603907

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1141.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006090914_marostegui_259292.log.

Kormat updated the task description. (Show Details)Jun 9 2020, 9:17 AM
Kormat added a subscriber: Kormat.

Removed the loan to labsdb from the description now that https://gerrit.wikimedia.org/r/603907 is merged.

Mentioned in SAL (#wikimedia-operations) [2020-06-09T09:29:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1148 to clone db1141 - T252512', diff saved to https://phabricator.wikimedia.org/P11423 and previous config saved to /var/cache/conftool/dbconfig/20200609-092915-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T09:34:55Z] <marostegui> Stop MySQL on db1148 to clone db1141 - T252512

Completed auto-reimage of hosts:

['db1141.eqiad.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-06-09T10:32:53Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1141 depooled to s4 T252512', diff saved to https://phabricator.wikimedia.org/P11425 and previous config saved to /var/cache/conftool/dbconfig/20200609-103252-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:14:43Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11426 and previous config saved to /var/cache/conftool/dbconfig/20200609-111443-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:27:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11427 and previous config saved to /var/cache/conftool/dbconfig/20200609-112701-marostegui.json

Change 603973 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1141: Enable notifications

https://gerrit.wikimedia.org/r/603973

Change 603973 merged by Marostegui:
[operations/puppet@production] db1141: Enable notifications

https://gerrit.wikimedia.org/r/603973

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:30:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11428 and previous config saved to /var/cache/conftool/dbconfig/20200609-113056-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:37:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11429 and previous config saved to /var/cache/conftool/dbconfig/20200609-113702-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:38:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11430 and previous config saved to /var/cache/conftool/dbconfig/20200609-113818-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:46:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1148 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11431 and previous config saved to /var/cache/conftool/dbconfig/20200609-114615-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T11:50:17Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11432 and previous config saved to /var/cache/conftool/dbconfig/20200609-115016-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-09T12:00:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1141 into s4 T252512', diff saved to https://phabricator.wikimedia.org/P11433 and previous config saved to /var/cache/conftool/dbconfig/20200609-120009-marostegui.json

Marostegui updated the task description. (Show Details)Jun 9 2020, 12:00 PM

@jcrespo can you take care of productionizing db1145? It will be the one to replace db1102 (whenever we feel comfortable having a backupsource running 10.4). For now I would like to have it installed, up and replicating, not necessarily performing backup options.

Apart from the above, I would also feel fine if you want to install it with Stretch+mariadb 10.1 so we can remove db1102 entirely and I can relocate it somewhere else (T253217)

Sorry, I hadn't seen this question, I will take care of this following discussion at T254871#6244800

jcrespo claimed this task.Jun 23 2020, 11:19 AM

About to move db1102 sections to db1145.

Change 607267 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install_server: Reimage and wipe db1145 into stretch

https://gerrit.wikimedia.org/r/607267

Change 607267 merged by Jcrespo:
[operations/puppet@production] install_server: Reimage and wipe db1145 into stretch

https://gerrit.wikimedia.org/r/607267

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1145.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006231240_jynus_20243.log.

Completed auto-reimage of hosts:

['db1145.eqiad.wmnet']

and were ALL successful.

Change 607288 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Setup db1145 as a mariadb backup source for s4, s5

https://gerrit.wikimedia.org/r/607288

Change 607288 merged by Jcrespo:
[operations/puppet@production] mariadb: Setup db1145 as a mariadb backup source for s4, s5

https://gerrit.wikimedia.org/r/607288

jcrespo updated the task description. (Show Details)Jun 24 2020, 12:52 PM

Change 607493 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Productionize db1145 after cloning db1102 into it

https://gerrit.wikimedia.org/r/607493

Assigning back to me to review what's pending after getting db1145 done by Jaime

Change 607493 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Productionize db1145 after cloning db1102 into it

https://gerrit.wikimedia.org/r/607493

So db1145 is done, loaded from db1102 snapshots (stretch), and functionality (backup sourcing) moved to it.

Assigning it to you so you can check what ahs been done and decide if to resolve or wait to check new backups occurring there successfully.

@jcrespo leaving notifications disabled in puppet for db1145 is intended? db1145 is green on icinga, so not sure what your plans are regarding notifications for that host.

Marostegui closed this task as Resolved.Jun 24 2020, 1:35 PM

All done - puppet pending to run and enable notifications on db1145

Marostegui changed the status of subtask T253217: Relocate "old" s4 hosts from Open to Stalled.Nov 6 2020, 9:35 AM