Page MenuHomePhabricator
Create Task
Maniphest T184397

Decommission db1030
Closed, ResolvedPublic
Actions

Description

In order to decommission db1030 first, the following movements are required:

  • In s8, make db1087 vslow server
  • Remove db1063 from s8 vslow and move it to s6 vslow
  • Clone db1063 from db1030
  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/406613/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/#/c/406981/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port , note on task for later unracking - asw-b-eqiad:ge-1/0/14
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system removed from rack

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 Resolved• CmjohnsonT184397 Decommission db1030

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 8 2018, 8:04 AM
• Marostegui triaged this task as Medium priority.Jan 8 2018, 8:05 AM
• Marostegui moved this task from Triage to Pending comment on the DBA board.
• Marostegui added a parent task: T134476: Decommission old coredb machines (<=db1050).
gerritbot subscribed.Jan 15 2018, 7:22 AM

Change 404258 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Replace db1063 with db1087 as vslow

https://gerrit.wikimedia.org/r/404258

gerritbot added a project: Patch-For-Review.Jan 15 2018, 7:22 AM
gerritbot added a comment.Jan 15 2018, 7:28 AM

Change 404258 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Replace db1063 with db1087 as vslow

https://gerrit.wikimedia.org/r/404258

• Marostegui added a subscriber: jcrespo.Jan 15 2018, 7:34 AM

I have set db1087 as vslow in s8 instead of db1063, which is the first step to move db1063 as vslow in s6, so we can get rid of db1030.

I will reimage db1063 as stretch and use 10.1 for it //cc @jcrespo

jcrespo added a comment.Jan 15 2018, 8:55 AM

Is is similar to https://gerrit.wikimedia.org/r/#/c/399792/3/wmf-config/db-eqiad.php or de we need an update?

• Marostegui added a comment.Jan 15 2018, 8:58 AM

It is exactly that :)

• Marostegui claimed this task.Jan 16 2018, 7:21 AM
• Marostegui moved this task from Pending comment to In progress on the DBA board.
gerritbot added a comment.Jan 17 2018, 3:48 PM

Change 404692 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1063.yaml: Disable notifications

https://gerrit.wikimedia.org/r/404692

gerritbot added a comment.Jan 17 2018, 3:49 PM

Change 404692 merged by Marostegui:
[operations/puppet@production] db1063.yaml: Disable notifications

https://gerrit.wikimedia.org/r/404692

gerritbot added a comment.Jan 22 2018, 6:27 AM

Change 405672 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Move db1063 to s6

https://gerrit.wikimedia.org/r/405672

gerritbot added a comment.Jan 22 2018, 6:39 AM

Change 405672 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Move db1063 to s6

https://gerrit.wikimedia.org/r/405672

Stashbot subscribed.Jan 22 2018, 6:41 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-22T06:41:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Move db1063 from s8 to s6 - T184397 (duration: 00m 58s)

gerritbot added a comment.Jan 22 2018, 6:49 AM

Change 405673 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1063 to s6

https://gerrit.wikimedia.org/r/405673

gerritbot added a comment.Jan 22 2018, 6:56 AM

Change 405673 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1063 to s6

https://gerrit.wikimedia.org/r/405673

ops-monitoring-bot subscribed.Jan 22 2018, 6:57 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db1063.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201801220657_marostegui_4121.log.

ops-monitoring-bot added a comment.Jan 22 2018, 8:43 AM

Completed auto-reimage of hosts:

['db1063.eqiad.wmnet']

Of which those FAILED:

['db1063.eqiad.wmnet']
ops-monitoring-bot added a comment.Jan 22 2018, 8:43 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db1063.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201801220843_marostegui_11772.log.

ops-monitoring-bot added a comment.Jan 22 2018, 8:44 AM

Completed auto-reimage of hosts:

['db1063.eqiad.wmnet']

Of which those FAILED:

['db1063.eqiad.wmnet']
ops-monitoring-bot added a comment.Jan 22 2018, 8:44 AM

Script wmf-auto-reimage was launched by marostegui on neodymium.eqiad.wmnet for hosts:

['db1063.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201801220844_marostegui_11978.log.

ops-monitoring-bot added a comment.Jan 22 2018, 9:04 AM

Completed auto-reimage of hosts:

['db1063.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jan 22 2018, 9:14 AM

Change 405681 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1030

https://gerrit.wikimedia.org/r/405681

gerritbot added a comment.Jan 22 2018, 9:19 AM

Change 405681 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1030

https://gerrit.wikimedia.org/r/405681

Stashbot added a comment.Jan 22 2018, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-22T09:20:46Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1030 - T184397 (duration: 00m 56s)

Stashbot added a comment.Jan 22 2018, 9:21 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-22T09:21:08Z] <marostegui> Stop MySQL on db1030 to clone db1063 - T184397

gerritbot added a comment.Jan 22 2018, 10:46 AM

Change 405688 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s6,s8.hosts: Move db1063 to s6

https://gerrit.wikimedia.org/r/405688

gerritbot added a comment.Jan 22 2018, 10:49 AM

Change 405688 merged by jenkins-bot:
[operations/software@master] s6,s8.hosts: Move db1063 to s6

https://gerrit.wikimedia.org/r/405688

gerritbot added a comment.Jan 22 2018, 12:07 PM

Change 405695 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Disable notifications db1030

https://gerrit.wikimedia.org/r/405695

gerritbot added a comment.Jan 22 2018, 12:10 PM

Change 405696 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1063 as vslow in s6

https://gerrit.wikimedia.org/r/405696

gerritbot added a comment.Jan 22 2018, 12:13 PM

Change 405695 merged by Marostegui:
[operations/puppet@production] mariadb: Disable notifications db1030

https://gerrit.wikimedia.org/r/405695

gerritbot added a comment.Jan 22 2018, 12:29 PM

Change 405696 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1063 as vslow in s6

https://gerrit.wikimedia.org/r/405696

Stashbot added a comment.Jan 22 2018, 12:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-22T12:30:49Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1063 as vslow - T184397 (duration: 00m 56s)

• Marostegui added a comment.Jan 22 2018, 12:31 PM

db1030 is no longer serving vslow in s6.
db1063 is now serving vslow there, let's leave it running for a week before proceeding and starting db1030 decommissioning process.

• Marostegui updated the task description. (Show Details)Jan 26 2018, 11:38 PM
gerritbot added a comment.Jan 29 2018, 5:52 PM

Change 406613 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1030

https://gerrit.wikimedia.org/r/406613

gerritbot added a comment.Jan 30 2018, 6:24 PM

Change 406843 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s6.hosts: Remove db1030

https://gerrit.wikimedia.org/r/406843

gerritbot added a comment.Jan 31 2018, 7:08 AM

Change 406613 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1030

https://gerrit.wikimedia.org/r/406613

Stashbot added a comment.Jan 31 2018, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-31T07:10:20Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1030, will be decommissioned - T184397 (duration: 00m 57s)

Stashbot added a comment.Jan 31 2018, 7:13 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-31T07:13:06Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1030, will be decommissioned - T184397 (duration: 00m 56s)

• Marostegui updated the task description. (Show Details)Jan 31 2018, 7:21 AM
gerritbot added a comment.Jan 31 2018, 7:27 AM

Change 406981 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1030

https://gerrit.wikimedia.org/r/406981

gerritbot added a comment.Jan 31 2018, 7:38 AM

Change 406981 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1030

https://gerrit.wikimedia.org/r/406981

gerritbot added a comment.Jan 31 2018, 7:42 AM

Change 406843 merged by jenkins-bot:
[operations/software@master] s6.hosts: Remove db1030

https://gerrit.wikimedia.org/r/406843

• Marostegui updated the task description. (Show Details)Jan 31 2018, 7:46 AM
• Marostegui updated the task description. (Show Details)
Stashbot added a comment.Jan 31 2018, 7:47 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-31T07:47:34Z] <marostegui> Remove db1030 from tendril - T184397

Stashbot added a comment.Jan 31 2018, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-31T07:48:38Z] <marostegui> Stop MySQL on db1030 - T184397

• Marostegui reassigned this task from • Marostegui to • Cmjohnson.Jan 31 2018, 8:06 AM
• Marostegui added a project: ops-eqiad.
• Marostegui moved this task from In progress to Done on the DBA board.
• Marostegui added a subscriber: • Cmjohnson.

db1030 is now ready to be fully decommissioned by @Cmjohnson

Restricted Application added a project: SRE. · View Herald TranscriptJan 31 2018, 8:06 AM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Feb 2 2018, 4:40 PM
• Marostegui added a project: hardware-requests.Feb 5 2018, 5:31 PM
RobH claimed this task.Feb 9 2018, 10:12 PM
gerritbot added a comment.Feb 9 2018, 10:15 PM

Change 409451 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] db1030 decom

https://gerrit.wikimedia.org/r/409451

gerritbot added a comment.Feb 9 2018, 10:18 PM

Change 409451 merged by RobH:
[operations/dns@master] db1030 decom

https://gerrit.wikimedia.org/r/409451

gerritbot added a comment.Feb 9 2018, 10:20 PM

Change 409454 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom db1030

https://gerrit.wikimedia.org/r/409454

gerritbot added a comment.Feb 9 2018, 10:21 PM

Change 409454 merged by RobH:
[operations/puppet@production] decom db1030

https://gerrit.wikimedia.org/r/409454

RobH reassigned this task from RobH to • Cmjohnson.Feb 9 2018, 10:24 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH subscribed.
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Feb 9 2018, 10:27 PM
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Mar 28 2018, 5:45 PM
gerritbot added a comment.Mar 28 2018, 6:18 PM

Change 422451 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns db1030

https://gerrit.wikimedia.org/r/422451

gerritbot added a project: Patch-For-Review.Mar 28 2018, 6:18 PM

Change 422451 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns db1030

https://gerrit.wikimedia.org/r/422451

• Cmjohnson updated the task description. (Show Details)Mar 28 2018, 6:19 PM
• Cmjohnson closed this task as Resolved.Apr 3 2018, 6:41 PM
• 238482n375 removed • Cmjohnson as the assignee of this task.Jun 15 2018, 8:01 AM
• 238482n375 lowered the priority of this task from Medium to Lowest.
• 238482n375 moved this task from Next Up to In Code Review on the Analytics-Kanban board.
• 238482n375 added projects: Analytics-Kanban, acl*security, Wikimedia-VE-Campaigns (S2-2018), Scap (Scap3-Adoption-Phase2), AbuseFilter, Data-release, Hashtags, LabsDB-Auditor, Ladies-That-FOSS-MediaWiki, Language-2018-Apr-June, Language-2018-Jan-Mar, HHVM, HAWelcome.
• 238482n375 edited subscribers, added: • 238482n375; removed: Aklapper.
This comment was removed by Reedy.
• 238482n375 set Security to Software security bug.Jun 15 2018, 8:05 AM
• 238482n375 changed the visibility from "Public (No Login Required)" to "Custom Policy".
This comment was removed by Reedy.
Reedy removed projects: HAWelcome, HHVM, Language-2018-Jan-Mar, Language-2018-Apr-June, Ladies-That-FOSS-MediaWiki, LabsDB-Auditor, Hashtags, Data-release, AbuseFilter, Scap (Scap3-Adoption-Phase2), Wikimedia-VE-Campaigns (S2-2018), acl*security, Analytics-Kanban, Patch-For-Review.Jun 15 2018, 1:52 PM
Reedy changed the visibility from "Custom Policy" to "Public (No Login Required)".
Reedy removed a subscriber: • 238482n375.
Restricted Application added a project: acl*security. · View Herald TranscriptJun 15 2018, 1:52 PM
Reedy removed a project: acl*security.Jun 15 2018, 1:52 PM
Reedy subscribed.
Reedy unsubscribed.
Aklapper raised the priority of this task from Lowest to Medium.Jun 15 2018, 2:05 PM
Aklapper subscribed.
Aklapper assigned this task to • Cmjohnson.Jun 15 2018, 2:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL