Page MenuHomePhabricator
Create Task
Maniphest T184090

Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029
Closed, ResolvedPublic
Actions

Description

For core hosts lower than db2030 that used to be masters (not misc hosts, which will be handled on a separate ticket):

  • db2016
  • db2017
  • db2018
  • db2019
  • db2023
  • db2028
  • db2029
  • Ops steps
    • All system services confirmed offline from production use (MySQL is now stopped)
    • Set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
    • Remove system from all lvs/pybal active configuration
    • Any service group puppet/hiera/dsh config removed
    • Update site.pp with role::spare::system
  • non interrupt steps
    • Disable puppet on host
    • Remove all remaining puppet references (include role::spare:system)
    • Power down host
    • Disable switch port
    • Switch port assignment noted on this task (for later removal)
    • Remove production dns entries
    • Puppet node clean, puppet node deactivate
  • final decom steps
    • System disks wiped (by onsite)
    • System unracked and decommissioned (by onsite), update racktables with result
    • Switch port configration removed from switch once system is unracked.
    • Mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Decom: Remove mgmt DNS entries for db201[6-9],db2023 and db202[8-9]operations/dnsmaster+3 -28
removing dns entries for db20(1[6-9]|2[389]operations/dnsmaster+3 -14
Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029operations/puppetproduction+0 -72
mariadb: Decommission old codfw mastersoperations/puppetproduction+30 -24
mariadb: Decommission old codfw mastersoperations/mediawiki-configmaster+1 -19
mariadb: Remove comments about partitioning on db2039operations/mediawiki-configmaster+1 -1
mariadb: Decommission db2028 - set as spareoperations/puppetproduction+5 -3
mariadb: Decommission db2028- retire from mediawiki configoperations/mediawiki-configmaster+0 -3
dblist: Remove db2028 for decommissionoperations/softwaremaster+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo triaged this task as Medium priority.Jan 3 2018, 5:50 PM
jcrespo created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2018, 5:50 PM
gerritbot subscribed.Jan 3 2018, 5:54 PM

Change 401765 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Decommission db2028- retire from mediawiki config

https://gerrit.wikimedia.org/r/401765

gerritbot added a project: Patch-For-Review.Jan 3 2018, 5:54 PM
gerritbot added a comment.Jan 3 2018, 5:59 PM

Change 401766 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Decommission db2028 - set as spare

https://gerrit.wikimedia.org/r/401766

gerritbot added a comment.Jan 3 2018, 6:02 PM

Change 401767 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dblist: Remove db2028 for decommission

https://gerrit.wikimedia.org/r/401767

Marostegui added a comment.Jan 3 2018, 6:10 PM

s6 was already checksummed as far as I remember

gerritbot added a comment.Jan 3 2018, 6:10 PM

Change 401767 merged by Jcrespo:
[operations/software@master] dblist: Remove db2028 for decommission

https://gerrit.wikimedia.org/r/401767

jcrespo added a comment.Jan 3 2018, 6:13 PM

s6 was already checksummed as far as I remember

That is correct: T160509 but who says I didn't break it since then, specially on codfw?

Marostegui added a comment.Jan 3 2018, 6:15 PM
In T184090#3872319, @jcrespo wrote:

s6 was already checksummed as far as I remember

That is correct: T160509 but who says I didn't break it since then, specially on codfw?

Always a possibility yes :)

gerritbot added a comment.Jan 3 2018, 6:15 PM

Change 401765 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Decommission db2028- retire from mediawiki config

https://gerrit.wikimedia.org/r/401765

gerritbot added a comment.Jan 3 2018, 6:23 PM

Change 401766 merged by Jcrespo:
[operations/puppet@production] mariadb: Decommission db2028 - set as spare

https://gerrit.wikimedia.org/r/401766

Marostegui moved this task from Triage to Pending comment on the DBA board.Jan 5 2018, 6:35 AM
gerritbot added a comment.Jan 6 2018, 7:37 PM

Change 402599 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Remove comments about partitioning on db2039

https://gerrit.wikimedia.org/r/402599

gerritbot added a comment.Jan 8 2018, 3:16 PM

Change 402599 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Remove comments about partitioning on db2039

https://gerrit.wikimedia.org/r/402599

jcrespo renamed this task from Decommission db2028 to Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029.Jan 15 2018, 5:18 PM
jcrespo claimed this task.
jcrespo removed a project: Patch-For-Review.
jcrespo updated the task description. (Show Details)
jcrespo moved this task from Pending comment to In progress on the DBA board.
jcrespo mentioned this in T169501: Move some masters away from B6.Jan 16 2018, 4:08 PM
jcrespo added a comment.Jan 17 2018, 11:45 AM

I am performing a quick check on these hosts (and at the same time, testing and improving compare.py) to double check there is no data loss before decommissioning them. I am only checking:

archive ar_id
page page_id
revision rev_id
text old_id
user user_id

plus wb_terms and image/old_image on wikidata en commonswiki

jcrespo mentioned this in T154485: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038).Jan 17 2018, 12:20 PM
gerritbot added a comment.Jan 19 2018, 11:07 AM

Change 405270 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Decommission old codfw masters

https://gerrit.wikimedia.org/r/405270

gerritbot added a project: Patch-For-Review.Jan 19 2018, 11:07 AM
gerritbot added a comment.Jan 19 2018, 11:14 AM

Change 405270 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Decommission old codfw masters

https://gerrit.wikimedia.org/r/405270

jcrespo updated the task description. (Show Details)Jan 19 2018, 11:31 AM
jcrespo updated the task description. (Show Details)Jan 19 2018, 11:35 AM
gerritbot added a comment.Jan 19 2018, 11:41 AM

Change 405273 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Decommission old codfw masters

https://gerrit.wikimedia.org/r/405273

gerritbot added a comment.Jan 19 2018, 3:42 PM

Change 405273 merged by Jcrespo:
[operations/puppet@production] mariadb: Decommission old codfw masters

https://gerrit.wikimedia.org/r/405273

Stashbot subscribed.Jan 19 2018, 5:11 PM

Mentioned in SAL (#wikimedia-operations) [2018-01-19T17:11:19Z] <jynus> stopping mariadb on db2016,17,18,19,23,28&29 T184090

jcrespo removed a project: Patch-For-Review.Jan 19 2018, 5:36 PM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)Jan 19 2018, 5:40 PM
jcrespo reassigned this task from jcrespo to Papaul.Jan 19 2018, 5:42 PM
jcrespo moved this task from In progress to Blocked external/Not db team on the DBA board.
jcrespo added a project: ops-codfw.

Papaul, these 7 old hosts are ready to go, and we should make room for others.

Restricted Application added a project: SRE. · View Herald TranscriptJan 19 2018, 5:42 PM
Papaul reassigned this task from Papaul to jcrespo.Jan 19 2018, 6:07 PM
Papaul subscribed.

@jcrespo thanks.

Can you please do the steps below and assign the task back to me. Thanks

Disable puppet on host
Remove all remaining puppet references (include role::spare:system)
Puppet node clean, puppet node deactivate

RobH claimed this task.Jan 19 2018, 6:10 PM
RobH subscribed.

Please note that we shoudl do those steps, not Jaime, since he cannot disable the switch port (which has to be done at the same time.)

I'll claim this and handle those steps.

jcrespo added a comment.Jan 19 2018, 6:13 PM

Note I have not a problem to do those if told, but specially disabling puppet should be done just before literally shutting down the servers and disconnecting then to avoid not receiving security upgrades, etc., so I prefer if robh can coordinate that (no matter who actually does it in the end.

RobH added a project: hardware-requests.Jan 19 2018, 6:19 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.
gerritbot added a comment.Jan 19 2018, 6:25 PM

Change 405340 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029

https://gerrit.wikimedia.org/r/405340

gerritbot added a project: Patch-For-Review.Jan 19 2018, 6:25 PM
gerritbot added a comment.Jan 19 2018, 6:29 PM

Change 405342 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] removing dns entries for db20(1[6-9]|2[389]

https://gerrit.wikimedia.org/r/405342

RobH added a comment.Jan 19 2018, 6:37 PM

Switch ports for later removal (once they are unracked):

ge-6/0/0 - db2016
ge-6/0/1 - db2017
ge-6/0/2 - db2018
ge-6/0/3 - db2019
ge-6/0/7 - db2023
ge-6/0/11 - db2028
ge-6/0/12 - db2029

RobH updated the task description. (Show Details)Jan 19 2018, 6:39 PM
gerritbot added a comment.Jan 19 2018, 6:40 PM

Change 405340 merged by RobH:
[operations/puppet@production] Decommission db2016, db2017, db2018, db2019, db2023, db2028, db2029

https://gerrit.wikimedia.org/r/405340

gerritbot added a comment.Jan 19 2018, 6:41 PM

Change 405342 merged by RobH:
[operations/dns@master] removing dns entries for db20(1[6-9]|2[389]

https://gerrit.wikimedia.org/r/405342

jcrespo moved this task from Blocked external/Not db team to Done on the DBA board.Jan 19 2018, 6:42 PM
RobH reassigned this task from RobH to Papaul.Jan 19 2018, 6:44 PM
RobH updated the task description. (Show Details)

Ok, these are all ready to have disks wiped, unracked, and racktables updated. Then feel free to assign back to me to complete the last two steps, thanks!

RobH removed a project: Patch-For-Review.Jan 19 2018, 6:45 PM
Papaul updated the task description. (Show Details)Jan 23 2018, 5:47 PM
Papaul updated the task description. (Show Details)Jan 30 2018, 5:58 PM
gerritbot added a comment.Jan 31 2018, 11:34 PM

Change 407173 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] Decom: Remove mgmt DNS entries for db201[6-9],db2023 and db202[8-9]

https://gerrit.wikimedia.org/r/407173

gerritbot added a project: Patch-For-Review.Jan 31 2018, 11:34 PM
Dzahn reassigned this task from Papaul to RobH.Feb 1 2018, 3:33 PM
gerritbot added a comment.Feb 12 2018, 5:11 PM

Change 407173 merged by Dzahn:
[operations/dns@master] Decom: Remove mgmt DNS entries for db201[6-9],db2023 and db202[8-9]

https://gerrit.wikimedia.org/r/407173

RobH closed this task as Resolved.Feb 12 2018, 5:13 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
jcrespo awarded a token.Feb 12 2018, 5:14 PM
RobH updated the task description. (Show Details)Feb 12 2018, 5:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL