Page MenuHomePhabricator
Create Task
Maniphest T176311

decommission db1036
Closed, ResolvedPublic
Actions

Description

It is getting low on disk space.

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/381233/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Set to spare: https://gerrit.wikimedia.org/r/#/c/381721/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to decom rack (by onsite)

Details

SubjectRepoBranchLines +/-
s2.hosts: Remove db1036operations/softwaremaster+0 -1
mariadb: Remove db1036 for decommissioningoperations/puppetproduction+6 -11
db-eqiad,db-codfw.php: Remove db1036operations/mediawiki-configmaster+0 -8
db-eqiad.php: Increase weight for db1101operations/mediawiki-configmaster+5 -5
db-eqiad.php: Depool db1036, fully repool db1101operations/mediawiki-configmaster+6 -6
db-eqiad.php: Increase weight for db1101operations/mediawiki-configmaster+5 -5
Pool db1101 as recentchanges replica for s2 with low weightoperations/mediawiki-configmaster+11 -6
mariadb: Depool db1101 for maintenanceoperations/mediawiki-configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Sep 20 2017, 11:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2017, 11:26 AM
gerritbot subscribed.Sep 20 2017, 11:28 AM

Change 379212 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1101 for maintenance

https://gerrit.wikimedia.org/r/379212

gerritbot added a project: Patch-For-Review.Sep 20 2017, 11:28 AM
gerritbot added a comment.Sep 20 2017, 11:43 AM

Change 379212 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1101 for maintenance

https://gerrit.wikimedia.org/r/379212

Stashbot subscribed.Sep 20 2017, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-20T11:49:33Z] <jynus> stopping replication on db1101 for faster repartition work T176311

jcrespo added a comment.Sep 20 2017, 11:52 AM

Repartitioning db1101 is ongoing (while replication is down) so that it can substitute db1036 role.

jcrespo added a comment.Sep 20 2017, 9:26 PM

partitioning finished, db1101 should be ready to be pooled as the new special slave.

gerritbot added a comment.Sep 22 2017, 1:30 PM

Change 379756 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] Pool db1101 as recentchanges replica for s2 with low weight

https://gerrit.wikimedia.org/r/379756

gerritbot added a comment.Sep 22 2017, 2:46 PM

Change 379756 merged by jenkins-bot:
[operations/mediawiki-config@master] Pool db1101 as recentchanges replica for s2 with low weight

https://gerrit.wikimedia.org/r/379756

• Marostegui moved this task from Triage to Pending comment on the DBA board.Sep 25 2017, 4:35 AM
gerritbot added a comment.Sep 25 2017, 7:48 AM

Change 380440 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1101

https://gerrit.wikimedia.org/r/380440

• Marostegui updated the task description. (Show Details)Sep 25 2017, 7:49 AM
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Sep 25 2017, 7:54 AM

Change 380440 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1101

https://gerrit.wikimedia.org/r/380440

Stashbot added a comment.Sep 25 2017, 7:56 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-25T07:56:04Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)

gerritbot added a comment.Sep 25 2017, 8:21 AM

Change 380444 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Increase weight for db1101

https://gerrit.wikimedia.org/r/380444

gerritbot added a comment.Sep 25 2017, 8:58 AM

Change 380448 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1036

https://gerrit.wikimedia.org/r/380448

gerritbot added a comment.Sep 25 2017, 9:04 AM

Change 380448 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1036, fully repool db1101

https://gerrit.wikimedia.org/r/380448

Stashbot added a comment.Sep 25 2017, 9:05 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-25T09:05:31Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)

Stashbot added a comment.Sep 25 2017, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2017-09-25T09:07:56Z] <marostegui> Add 50GB to db1036 /srv partition - T176311

• Marostegui claimed this task.Sep 25 2017, 9:09 AM
• Marostegui moved this task from Pending comment to In progress on the DBA board.

db1036 is now fully depooled and db1101 is serving as a special slave.
Let's give it a couple of days and then proceed to get rid of db1036 from all the config files.

Meanwhile I have added 50GB to /srv to avoid it showing up on icinga (it was 91% filled and now 88%)

gerritbot added a comment.Sep 25 2017, 9:21 AM

Change 380444 abandoned by Marostegui:
db-eqiad.php: Increase weight for db1101

Reason:
Done here: https://gerrit.wikimedia.org/r/#/c/380448/

https://gerrit.wikimedia.org/r/380444

gerritbot added a comment.Sep 28 2017, 3:47 PM

Change 381233 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1036

https://gerrit.wikimedia.org/r/381233

gerritbot added a comment.Oct 2 2017, 6:05 AM

Change 381233 merged by Marostegui:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1036

https://gerrit.wikimedia.org/r/381233

Stashbot added a comment.Oct 2 2017, 6:07 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-02T06:07:07Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 48s)

• Marostegui updated the task description. (Show Details)Oct 2 2017, 6:07 AM
Stashbot added a comment.Oct 2 2017, 6:08 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-02T06:08:10Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1036 from config files as it will be decommissioned - T176311 (duration: 00m 46s)

gerritbot added a comment.Oct 2 2017, 6:14 AM

Change 381721 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1036 for decommissioning

https://gerrit.wikimedia.org/r/381721

gerritbot added a comment.Oct 2 2017, 6:17 AM

Change 381721 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1036 for decommissioning

https://gerrit.wikimedia.org/r/381721

• Marostegui reassigned this task from • Marostegui to • Cmjohnson.Oct 2 2017, 6:38 AM
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from In progress to Done on the DBA board.
• Marostegui added a project: ops-eqiad.
• Marostegui added a subscriber: • Cmjohnson.

db1036 is now ready to be totally decommissioned by @Cmjohnson

gerritbot added a comment.Oct 2 2017, 7:41 AM

Change 381736 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s2.hosts: Remove db1036

https://gerrit.wikimedia.org/r/381736

Stashbot added a comment.Oct 2 2017, 7:41 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-02T07:41:53Z] <marostegui> Stop MySQL on db1036 as it is going to be decommissioned - https://phabricator.wikimedia.org/T176311

gerritbot added a comment.Oct 2 2017, 7:43 AM

Change 381736 merged by jenkins-bot:
[operations/software@master] s2.hosts: Remove db1036

https://gerrit.wikimedia.org/r/381736

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Oct 2 2017, 4:36 PM
• Marostegui mentioned this in T177405: rack and setup db1107 and db1108.Oct 4 2017, 5:35 PM
• Marostegui added a project: hardware-requests.Nov 7 2017, 9:16 AM
• Cmjohnson updated the task description. (Show Details)Nov 8 2017, 6:50 PM
• Cmjohnson updated the task description. (Show Details)Nov 8 2017, 8:36 PM
• Cmjohnson closed this task as Resolved.Nov 14 2017, 4:24 PM

wiped, racktables updated

• Marostegui mentioned this in T162699: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036).Dec 5 2017, 6:28 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL