Page MenuHomePhabricator

Relocate "old" s4 hosts
Open, MediumPublic

Description

The following hosts have been replaced in s4, which hosts with larger disks:

eqiad:

  • db1081
  • db1084 (will be back to backup testing test-s1?)
  • db1091 (moved to s1)
  • db1097 (moved to m1) T254556
  • db1102 (currently backup source > moved to x1)
  • db1103 (moved to x1, to replace db1127 so db1127 can go to s7)
  • db1121
  • db1138

codfw:

  • db2090
  • db2073
  • db2091 (moved to s8)
  • db2099
  • db2084 (moved to s8)

They should be relocated to other places including:

At least:

  • extra hosts in s1 DONE: db1091
  • extra hosts in s8 (including extra vslow): db2084, db2091
  • extra host in s7 DONE: db1127
  • Maybe replace x1 hosts with these ones (smaller disks) DONE: db1103
  • One host must go for backup testing to replace the one that was taken: db1084

Other movements

  • db1127 from x1 to s7
  • db1135 from m1 to s1 and db1080 to m2 (finally db1080 will be moved to m1 because of T256717)

Related Objects

StatusSubtypeAssignedTask
ResolvedCmjohnson
ResolvedMarostegui
OpenMarostegui

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2020-06-10T11:02:04Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094 moving to clone db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11453 and previous config saved to /var/cache/conftool/dbconfig/20200610-110204-marostegui.json

Change 604550 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1127: Enable notifications

https://gerrit.wikimedia.org/r/604550

Change 604550 merged by Marostegui:
[operations/puppet@production] db1127: Enable notifications

https://gerrit.wikimedia.org/r/604550

Mentioned in SAL (#wikimedia-operations) [2020-06-11T04:47:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1084 and slowly repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11462 and previous config saved to /var/cache/conftool/dbconfig/20200611-044725-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-11T05:04:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11465 and previous config saved to /var/cache/conftool/dbconfig/20200611-050446-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-11T05:25:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11466 and previous config saved to /var/cache/conftool/dbconfig/20200611-052535-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-11T05:55:36Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1127 T253217', diff saved to https://phabricator.wikimedia.org/P11467 and previous config saved to /var/cache/conftool/dbconfig/20200611-055536-marostegui.json

Marostegui updated the task description. (Show Details)Thu, Jun 11, 7:47 AM
Marostegui updated the task description. (Show Details)Fri, Jun 12, 7:40 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Fri, Jun 12, 10:15 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-12T11:14:22Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db2080 and db2084 into s8 T253217', diff saved to https://phabricator.wikimedia.org/P11481 and previous config saved to /var/cache/conftool/dbconfig/20200612-111422-marostegui.json

Change 605193 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2084: Enable notifications

https://gerrit.wikimedia.org/r/605193

Change 605193 merged by Marostegui:
[operations/puppet@production] db2084: Enable notifications

https://gerrit.wikimedia.org/r/605193

Marostegui updated the task description. (Show Details)Fri, Jun 12, 11:23 AM

Change 605199 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2099

https://gerrit.wikimedia.org/r/605199

Change 605199 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2099

https://gerrit.wikimedia.org/r/605199

Marostegui updated the task description. (Show Details)Mon, Jun 15, 12:57 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-15T12:58:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2091:3312, db2091:3314 - T253217', diff saved to https://phabricator.wikimedia.org/P11495 and previous config saved to /var/cache/conftool/dbconfig/20200615-125856-marostegui.json

Change 605586 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage db2091

https://gerrit.wikimedia.org/r/605586

Change 605586 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage db2091

https://gerrit.wikimedia.org/r/605586

Change 606149 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db2091 to s8

https://gerrit.wikimedia.org/r/606149

Change 606149 merged by Marostegui:
[operations/puppet@production] mariadb: Move db2091 to s8

https://gerrit.wikimedia.org/r/606149

Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts:

['db2091.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006171023_marostegui_4864.log.

Completed auto-reimage of hosts:

['db2091.codfw.wmnet']

and were ALL successful.

Mentioned in SAL (#wikimedia-operations) [2020-06-17T12:40:35Z] <marostegui@cumin2001> dbctl commit (dc=all): 'Add db2091 to s8 T253217', diff saved to https://phabricator.wikimedia.org/P11566 and previous config saved to /var/cache/conftool/dbconfig/20200617-124034-marostegui.json

Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts:

['db2091.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006171246_marostegui_25785.log.

Completed auto-reimage of hosts:

['db2091.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Wed, Jun 17, 1:18 PM

Change 606311 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2091: Enable notifications

https://gerrit.wikimedia.org/r/606311

Change 606311 merged by Marostegui:
[operations/puppet@production] db2091: Enable notifications

https://gerrit.wikimedia.org/r/606311

Marostegui updated the task description. (Show Details)Thu, Jun 18, 4:53 AM
Marostegui updated the task description. (Show Details)Mon, Jun 22, 2:36 PM
jcrespo updated the task description. (Show Details)Mon, Jun 22, 3:07 PM
Marostegui updated the task description. (Show Details)Mon, Jun 22, 3:07 PM
jcrespo updated the task description. (Show Details)Mon, Jun 22, 3:09 PM
jcrespo updated the task description. (Show Details)Mon, Jun 22, 3:13 PM
Marostegui updated the task description. (Show Details)Mon, Jun 22, 3:14 PM

I intend to "take" db1102, delete its data and setup x1 with buster on it to generate 10.4 backups.

I intend to "take" db1102, delete its data and setup x1 with buster on it to generate 10.4 backups.

Remember that you can also take db1084 anytime now (needs to be depooled first)

Marostegui updated the task description. (Show Details)Thu, Jun 25, 4:47 AM
Marostegui updated the task description. (Show Details)Thu, Jun 25, 5:02 AM
Marostegui updated the task description. (Show Details)Thu, Jun 25, 5:05 AM

Change 607728 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1135: Disable notifications

https://gerrit.wikimedia.org/r/607728

Change 607728 merged by Marostegui:
[operations/puppet@production] db1135: Disable notifications

https://gerrit.wikimedia.org/r/607728

Marostegui updated the task description. (Show Details)Thu, Jun 25, 11:59 AM

Change 608256 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1135 to s1

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608256

Change 608256 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1135 to s1

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608256

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1135.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006290440_marostegui_27864.log.

Mentioned in SAL (#wikimedia-operations) [2020-06-29T04:57:08Z] <marostegui> Stop MySQL on db1080 to clone db1135 T253217

Completed auto-reimage of hosts:

['db1135.eqiad.wmnet']

and were ALL successful.

Change 608259 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] instances.yaml: Remove db1080, add db1135

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608259

Change 608259 merged by Marostegui:
[operations/puppet@production] instances.yaml: Remove db1080, add db1135

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608259

Marostegui updated the task description. (Show Details)Mon, Jun 29, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-29T07:46:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1135 (depooled) to s1 T253217', diff saved to https://phabricator.wikimedia.org/P11684 and previous config saved to /var/cache/conftool/dbconfig/20200629-074611-marostegui.json

Change 608265 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1135: Enable notifications

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608265

Change 608265 merged by Marostegui:
[operations/puppet@production] db1135: Enable notifications

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608265

Mentioned in SAL (#wikimedia-operations) [2020-06-29T08:02:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11685 and previous config saved to /var/cache/conftool/dbconfig/20200629-080253-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-29T08:26:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11686 and previous config saved to /var/cache/conftool/dbconfig/20200629-082635-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-29T08:36:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11687 and previous config saved to /var/cache/conftool/dbconfig/20200629-083631-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2020-06-29T08:48:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Fully pool db1135 into s1 T253217', diff saved to https://phabricator.wikimedia.org/P11688 and previous config saved to /var/cache/conftool/dbconfig/20200629-084827-marostegui.json

Change 608508 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1080 from s1 to m2

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608508

Change 608508 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1080 from s1 to m2

https://gerrit.wikimedia.org/r/c/operations/puppet/ /608508

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1080.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006300451_marostegui_8504.log.

Completed auto-reimage of hosts:

['db1080.eqiad.wmnet']

Of which those FAILED:

['db1080.eqiad.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1080.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006300504_marostegui_10430.log.

Completed auto-reimage of hosts:

['db1080.eqiad.wmnet']

Of which those FAILED:

['db1080.eqiad.wmnet']
Marostegui updated the task description. (Show Details)Tue, Jun 30, 7:59 AM
Marostegui updated the task description. (Show Details)Tue, Jun 30, 9:40 AM